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Introduction 


The  U.S.  Air  Force  Medical  Service  presented  the  sixth  annual  Air  Force  Medical  Research  Symposium  coordinated  by  the 
Air  Force  Medical  Support  Agency’s  Research  and  Development  Division  (AFMSA/SGRS).  The  symposium  was  held  on  2- 
4  August  2011  in  the  Washington  DC  area  at  the  Gaylord  National  Resort  Hotel  and  Convention  Center  in  National  Harbor, 
MD.  The  symposium  featured  two  half-days  of  plenary  sessions,  one  and  a  half  days  of  scientific  presentations,  and  a  poster 
session. 

The  symposium  was  organized  into  several  tracks  to  include  Enroute  Care,  Force  Health  Protection,  Healthcare  Informatics, 
Operational  Medicine  (In-Garrison  Care),  and  Psychological  Health/Traumatic  Brain  Injury,  as  follows: 

•  The  Enroute  Care  Track  addressed  science  and  technology  targeted  at  the  continuum  of  care  during  transport  from 
point  of  injury  to  definitive  care  including,  but  not  limited  to:  Casevac,  Medivac;  Aeromedical  Evacuation;  Critical 
Care  Air  Transport;  and  Patient  Staging.  Further  areas  addressed  included:  patient  stabilization;  patient  preparation 
for  movement;  impact  of  in-transit  environment  on  patient  and  AE  crew  physiology;  human  factors  concerns  for  AE 
crew  or  patient  population;  AE/medical  personnel  training;  infectious  disease/control;  bum  management;  pain 
management;  resuscitation;  lifesaving  interventions;  and  nutrition  research  in  the  enroute  care  environment. 

•  The  Force  Health  Protection  Track  focused  on  prevention  of  injury  and  illness  and  the  early  recognition  or  detection 

of  emerging  threats  for  in-garrison  or  deployed  operations.  Topics  of  interest  include  research  in  bio-surveillance, 
infectious  disease,  emerging  threats  (pandemic  response),  protective  countermeasures,  disaster 

response/consequence  management,  toxicology/health  risks  (e.g.,  particulates  nanomaterials,  radiation,  etc.), 
monitoring  disease  trends,  other  areas  of  preventive  medicine,  public  and  environmental  health  relevant  to  the 
military  workforce. 

•  The  Healthcare  Informatics  Track  focused  on  the  use  of  innovative  information  management  &  technology  solutions 
that  enhance  healthcare  delivery  at  any  point  of  the  full  spectmm  of  patient  care  to  include  medical  simulation  and 
training. 

•  The  Operational  Medicine  (In-Garrison  Care)  Track  focused  on  care  delivered  in  the  outpatient  or  inpatient  in¬ 
garrison  setting  and  on  enhancing  the  performance  of  airman  in  challenging  operational  and  expeditionary 
environments. 

•  The  Psychological  Health/Traumatic  Brain  Injury  Track  addressed  topics  pertaining  to  screening,  diagnosis,  and 
treatment  of  TBI  and/or  Psychological  Health  in  the  military  community.  Specific  focus  areas  within  Psychological 
Health  included  depression,  substance  use  disorders,  family  functioning,  and  suicide  prevention.  Topics  of  special 
interest  included  field-deployable  diagnostic  tests  for  mild  TBI  (concussion),  blast  modeling,  large  epidemiologic 
studies  of  Psychological  Health  and  TBI,  and  strategies  for  translating  research  into  practice. 

These  proceedings  are  organized  into  five  volumes,  as  follows: 

•  Volume  1.  This  volume  is  a  general  overview  of  the  entire  2011  Air  Force  Medical  Research  Symposium  and 
includes  abstracts  of  all  the  oral  presentations  and  posters.  First  presented  is  the  symposium’s  opening 
plenary  session,  followed  by  the  abstracts  from  the  four  technical  tracks,  and  then  the  closing  plenary 
session.  The  abstracts  associated  with  the  poster  session  are  in  the  last  section  of  these  proceedings.  The 
agenda  for  the  overall  symposium  is  in  Appendix  A,  attendees  are  listed  in  Appendix  B,  and  continuing 
education  information  is  in  Appendix  C  of  this  volume.  Appendices  D-J  are  copies  of  presentation 
slides  from  the  plenary  sessions. 

This  volume  contains  abstracts  and  presentation  slides  for  the  Enroute  Care  Track. 

This  volume  contains  abstracts  and  presentation  slides  for  the  Force  Health  Protection  Track. 

This  volume  contains  abstracts  and  presentation  slides  for  the  Healthcare  Informatics  Track. 

This  volume  contains  abstracts  and  presentation  slides  for  the  Operational  Medicine  (In-Garrison  Care) 
Track. 

This  volume  contains  abstracts  and  presentation  slides  for  the  Psychological  Health/Traumatic  Brain 
Injury  Track. 


•  Volume  2. 

•  Volume  3. 

•  Volume  4. 

•  Volume  5. 

•  Volume  6. 
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Patient-Centered  Precision  Care  (PC2) 

Dr.  Ronald  Miller,  SG9Z,  Air  Force  Medical  Support  Agency 

The  Air  Force  Patient-Centered  Precision  Care  (PC2Z)  Program  has  been  established  to  guide  the  use  of 
genomic  information  in  clinical  decision-making  as  the  field  of  personalized  medicine  advances  and  medical 
evidence  accumulates.  Recent  advances  in  genomic  technology  have  suggested  that  analyses  of  a  patient’s 
genome  can  provide  information  on  an  individual’s  health,  identifying  a  patient’s  response  to  medication  or  a 
person’s  risk  of  developing  disease  relative  to  the  average  population.  In  order  to  fully  realize  the  potential  of 
genomic  medicine,  further  work  must  be  done  to  demonstrate  its  clinical  utility  and  to  establish  an  effective 
infrastructure  for  the  integration  of  genomics  into  clinical  care.  To  achieve  these  goals,  the  PC2Z  Program  is 
composed  of  four  major  pillars: 

1 .  Policy:  to  identify  and  address  the  ethical,  legal,  and  social  issues  associated  with  the  utilization  of  genomic 
information  in  clinic. 

2.  Research:  to  longitudinally  assess  the  clinical  utility  of  the  genomic  information  in  the  delivery  of  health 
care.  Additionally,  de-identified  genomic  information  will  be  provided  to  the  government  and  academic 
partners  for  use  in  additional  genetic  studies  aimed  at  discovering  novel  disease-gene  associations. 

3.  Informatics:  to  evaluate  methods  for  the  storage,  protection,  and  integration  of  genomic  information  into 
the  existing  electronic  healthcare  records. 

4.  Education:  to  provide  educational  resources  for  medical  staff  and  patients  on  interpretation  and  benefits  of 
genomic  information  in  the  delivery  of  health  care. 

Through  the  PC2Z  program,  genomic  data  will  become  a  valuable  resource,  informing  the  efficient  and  targeted 
delivery  of  health  care  to  patients  in  the  future. 
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AFMS  Patient-Centered 
Precision  Care  (PC2Z) 

Genomic  Medicine  Research 

Chris  Bradburne,  PhD  and  Ruth  Vogel,  MPH 
The  Johns  Hopkins  University, 
Applied  Physics  Laboratory 
Major  Heather  Halvorson,  MD,  MPH 
Major  Cecili  Sessions,  MD,  MPH 
AFMS  Medical  Innovations  Division 
U.S.  AIR  FORCE  August  2011 


U.S.  AIR  FORCE 


■  Overview  Patient-Centered  Precision  Care  (PC2) 


Agenda 


■  Genomic  Disease  and  Medicine 

■  Implementing  Genomic-informed  Medicine 

■  PC2Z  Program  Approach  and  Objectives 

■  Knowledge  Generation  Research  Pillar 

■  Bioinformatics  Research  Pillar 

■  Genomic  Education  Research  Pillar 

■  Ethical,  Legal,  and  Social  Issues  /  Policy  Research  Pillar 

■  Systems  Engineering  /  Advance  Genomic  Diagnostic  System 


■  Whole  Genome  Sequencing  and  the  future  of  PC2Z 

■  Future  PC2Z  Program  Collaborative 
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PC2  Overview 


Genomic  Disease  and  Medicine 


ULS-  AIR  FORCE 


■  Air  Force  Medical  Service  Patient-Centered  Precision  Care  (PC2) 

■  State  of  the  art,  evidence-based,  personalized  care 
incorporating  all  available  patient  information 

■  Targeted  prevention,  diagnostics  and  therapy 

■  Two  parallel  efforts 

■  PC2-Clinical  (Air  Force  Clinical  Decision  Support) 

■  PC2-Z  (Genomic  Medicine  Research) 

■  PC2-Z  (Genomic  Medicine  Research) 

■  Advance  genome-informed  personalized  medicine  in  AFMS 

■  Leverage  existing  AFMS  resources,  infrastructure,  and  data 

■  Develop  agile,  scalable  program 
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■  The  long  road  to  realizing  genome-informed  medicine 


1990 

Structure  and  primary 
sequence  of  genomes 

2000 

Biology  and  variation 
of  genomes 


2011 

Biology  of 
disease 
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Genomic  Disease  and  Medicine 


Genomic  Disease  and  Medicine 
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Ranges  of  structural  variation  in  the  genome 
influence  inherited  phenotypes  and  conditions 


Whole  chromosome  to  whol*  genome 
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■  Inheritance  of  genetic  diseases 
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a  Complex  Conditions 

a  Multi  genic,  complex,  etc  —  GWAS  shews  associations  or 

markers  which  allow  for  risk  estimation 
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Implementing 
Genome-Informed  Medicine 

■  Utilizing  GWAS  data  to  screen  individuals  for  disease  conditions 
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>  Inflammatory  bowel  dfeoaso 
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v  Hypertension 

>  Rheumatoid  arthritis 

>  Human  height 
v  Others 
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Implementing 
Genome-Informed  Medicine 

■  Approach  to  personalized  medicine  must  include  not  o  nly  genomics 
■  PC1* -Clinical 
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For  any  condition  the  overall  balance  of  genetic  and  environmental 
determinants  can  be  represented  by  a  point  somewhere  within  the  triangle 
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Implementing 
u_jLSi  Genome-Informed  Medicine 

■  Collaboration  is  absolute!/  necessary 

■  Technology  outpacing  medical  erf  dance  lor  clinical  Implementation 

■  Perception:  ability  to  sequence  the  genome  -  clinical  action 

■  Complex  conditions?  Clinical  utility?  Modest  effect  size? 

■  Show -stopping  ethical,  legal,  and  social  issues  for  military 

■  Need  to  establish  policy  and  to  adapt  to  changes  in  fie  Id /'society 

■  Healthcare  teams  and  patients  need  education  and  experience 

■  Requires  proven,  robust  health  informatics  in  routine  clinical  use 

■  Solution  system  will  need  to  incorporate  regulatory  requirements, 
need  for  accepted  standards,  interoperability  across  health  systems, 
patient  rights,  intellectual  property/reimbursement  translation 
function  for  rapid  integration  of  new  evidence 

integrity  Service  Excellence 
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PC2Z  Program  Approach 


■  Consortium  of  leading  experts  to  direct  /  inform  integrated  program 


■  Government  and  academic  partners;  JHiU  APL  program  Integrator 


■  Collaboration  and  transparency 
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PC2Z  Program  Approach 


PC2Z  Pillars 
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m  Know  led  ge  Gene  ration  /  Re  s  e □  rch  Pi  I  tar  (KGR) 

■  Objective:  expand  evidence  for  clinical  utility  of  genomic 
information 


■  Longitudinal  Clinical  Utility  Study 

■  Report  genomic  risk  to  participants 

■  Complex  conditions  treated  in  Primary  Care 

■  Potentially  clinically  actionable 

■  Objective:  create  digital  blobank 

■  Full  sequence  and  cITnica!  data 

■  Objective:  support  knowledge  generation  i  discovery  studies 

■  NIH  NCBk  dbGAP  mechanism 
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CUS  Overview 


PC2Z  Pillars 


Clinica[  Utility  Study 

Sharing 

Results 


Collects 

|  Medical  History  | 


Family  History  | 

Lifestyle  | 


Tests  for 

|  Chronic  Diseases"] 

[  Drug  Sensitivity  | 


Informed  Account 

Consents  Activations 

Saliva  Health  Genetic 

Collection  Questionnaires  Testing 


Risk 

Results 


Outcomes 

Research 


Reports 


|  Non-genetic  risk 
|  Genetic  risk  ~| 

I  Educational  I 

materials 
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PC2Z  Pillars 
Clinical  UtMty  Study 


CUS  Outcomes  Research: 


Risk  | 
Results 


Longitudinal  survey  data  collection 


w 

PC2Z  Pillars  W 

PC2Z  Pillars 

U.S.  AIR  FORCE 

Clinical  Utility  Study 

Clinical  Utility  Study 

■  Preliminary  results  on  participation  from  the  CPMC. 

■  88%  opted  to  allow  release  of  de-identified  genetic  and  phenotypic 
data  to  nonprofit  entities 


■  Participants  who  complete  questionnaires  are  likely  to  view  results 


1  Participant  Description  n  1  %  relative  to 

%  relative  to  total  that 

%  relative  to  total 

total 

activate  web  accounts 

that  complete 

enrolled 

questionnaires 

Total  enrolled  fror| 

Reasons  for  requesting  genetic  counseling 

2009 

Those  with  active 

Understanding  risk 

5.9% 

account  Complex  Disease  Genetics  11.3% 

Parti  al  compl  etior  Wh  at  d  o  I  d  o  now?  14.7% 

based  question  ns 

All  web-based  Basic  genetics  15.1% 

questionnaires  co  other  (requests  for  results  status,  future  results,  or  52.1  % 

Customized  risk  r  other  info  outside  the  scope  of  the  study) 
available  on  web 

Risk  report  viewed  by  1674  38  52  87 

participant  Data  from  Keller  et.  al.,  Personalized  Medicine,  2010 


■  Condition  selection  and  risk  estimation  is  done  via  a  rigorous, 
published  literature  curation  process  and  oversight  boards 
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PC2Z  Pillars 
Utility  Study 

■  Current,  board-approved  conditions  and  drug/variant  pairs 

!  ICOB/PAG  approved  conditions 

Risk  factors 

|  (other  than  genetic  varian 

1  Multi -variable  adjustment 

0 _ _ __ _ ■ 

Age- related  macular  degeneration 

Family  history 

Smoking 

None 

Coronary  Artery  Disease 

Family  history 
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Diabetes 
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Iron  Overload  /  Hemochromatosis 
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Type 2  Diabetes 
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Body  Mass  Index 

Yes 

Melanoma 

Family  history 

None 

Prostate  Cancer 

Family  history 

Yes 

Lupus 

Family  history 

Smoking 

None 

Type  1  Diabetes 

Family  history 

Yes 

CYP2C19 

Cbpidogrel 

Use  of  Proton  pump  inhibitors 

None 
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Risk  reporting  is  not  only  genetic  risk , 
but  afi  known  risk  factors 
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PC2Z  Pillars 
Bioinforma  tics 


■  Objective:  develop  learning  system  that  incorporates  genomic  data 
into  existing  and  future  DoD  informatics  tools 


System  Adaptability 


Architecture  of  AF  Medical 
Informatics  System  must  be 
flexible  and  adaptable  to 
accommodate  changing 
information  requirements 
resulting  from  changing 
medical  knowledge  and 
practice. 


S 

Perec  nalized 
Medicine 
Policies 

ir 

Information 

Requirements 

* 
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Informatics 

System 


Bio  informatics 
and  medical 
research 
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De-ldentified 
Medical  Record 
Databases 


Genomics 
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PC2Z  Pillars 
Education  Research  Pillar 


■  Objective:  ensure  healthcare  team,  to  include  the  patient,  has  an 
adequate  understanding  of  genomic  medicine 


Integrity  -  Service  -  Excellence 
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W  PC2Z  Pillars 

u  3 ELS//  Policy  Research  PUIar 

■  Objective:  develop  agilo,  adaptable  mechanism  to  enable  AF/OoO 
policy  that  is  evidence-based  and  current 

■  Objective:  address  ELSi  issues  throughout  program 


Program  Needs  to  be  Anticipatory,  Not  Reactive 
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PC2Z  Pillars 


a  Systems  Engineering  /  Advanced  Genomic  Diagnostic  Concepts 
■  Objective:  develop  diagnostic  system  which  enables  delivery 
of  full-sequence  genomic  information  Into  health  system 
workflow 

■  integrate  expanding  evidence-base 

■  informatics  standards  for  interoperability 
across  diverse  platforms 

■  ethical,  legal,  and  social  issues 

■  patient  autonomy/rights 

■  regulatory,  intellectual  property, 
medical-legal  documentation  and 
reimbursement  requirements 


Integrity -  Service  -  Excellence 


The  Future  -  Whole  Genome 


Sequencing 

'■■  ■  ■  ■  I 


Cost  pci  Human  Genuine 


Future  PC2Z  Collaborative 


a  Interested  AFMS  researchers 


■  Interested  parties  for  ancillary  studies 
e  Interested  partners  from  other  DOD  medical  services 
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Patient  Health  Record  with  Secure  Messaging  (PHR/SM)  Implementation  at  Elmendorf  AFB 
Drs.  Ritu  Agarwal  and  Catherine  Anderson,  University  of  Maryland 

We  describe  early  results  of  a  pilot  patient  health  record  (PHR)  project  implemented  and  deployed  at  Elmendorf 
Air  Force  Base  in  December  2010.  The  PHR  tool  supports  entry  and  management  of  health  information 
directly  by  patients,  integrates  with  the  patient’s  clinical  record,  and  supports  secure  patient-provider 
messaging.  It  is  a  core  component  of  the  US  Air  Force’s  transition  towards  a  healthcare  delivery  system  that  is 
patient-focused  and  incorporates  principles  of  the  Patient  Centered  Medical  Home.  We  provide  a  brief 
summary  of  the  project  from  its  initial  motivation  through  development  and  the  go-live  period.  We  outline  our 
long-term  research  goal  which  is  to  gather  evidence  to  demonstrate  the  value  of  this  suite  of  tools  on  patients’ 
health  outcomes,  their  empowerment  in  making  health-related  decisions,  engagement  with  healthcare,  and  the 
efficiency  of  health  services  delivery.  Finally,  we  provide  early  evidence  from  surveys  of  users  and  providers 
conducted  at  the  launch  of  the  pilot  to  assess  their  baseline  expectations  about  the  system  and  insights  on 
effectiveness  of  change  management  efforts.  1,639  patients  registered  during  the  project’s  three  month  baseline 
period.  283  patients  responded  to  the  email  survey  requests.  Approximately  half  of  the  providers  completed 
surveys.  While  it  is  very  early  in  the  implementation  of  the  PHR  and  available  data  for  analysis  is  limited,  we 
are  able  to  make  a  few  recommendations  based  on  preliminary  findings.  Early  results  indicate  an 
overwhelmingly  positive  patient  response  to  the  PHR  tool  which  is  not  reflected  to  the  same  degree  by  the 
providers.  Consequently,  training  and  messaging  targeted  toward  providers  should  be  positive  but  also 
realistically  set  expectations.  As  the  PHR  is  deployed  at  other  MTFs,  opportunities  to  personally  promote  the 
PHR  via  registration  desks  and  directly  through  providers  and  staff  should  be  emphasized  as  findings  suggest 
these  mechanisms  result  in  higher  positive  patient  perceptions  of  and  intentions  to  use  the  PHR. 
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Agenda 


■  Empowering  patients  with  technology 

■  Project  background  &  objectives 

■  Project  inception  and  evolution 

■  Project  organization 

■  System  rollout 

■  Early  adopters  -  who  are  they  and  what  are  their  usage 
patterns? 

■  Research  study  overview  &  baseline  highlights 

■  Next  steps 


Integrity  -  Service-  Excellence 


Empowering  Patients  with  Technology 


U.S.  AIR  FORCE 


U.S.  AIRFORCE 


Project  Background  &  Objectives 


WebMD  ■ 

(tatter  infnrmaftnn  Ruttcv  health  * 


Consumers  are  interested  in  self 
management  of  health  info  (Pew 
2011:  Deloitte  2008) 

91  viable’  PHR  products  with 
different  features  and 
characteristics  (Jones  et  al  2010) 

Despite  importance  of  PHR  tools 
for  patient  engagement  and 
potential  benefits  in  cost  reduction 
and  improved  quality  of  care,  there 
exists  limited  evidence  about  PHR 
deployment  and  use 


■  AFMS  key  aims  are  to  improve  overall  readiness,  experience  of  care, 
population  health  and  per  capita  cost 

■  AFMS  is  committed  to  pursuing  strategies  consistent  with  the 
principles  of  the  patient-centered  medical  home  (PCMH)  initiative 


■  Goals  of  the  PHR/SM  implementation  project: 

■  Improve  quality  of  healthcare 

■  Increase  staff  productivity 

■  Decrease  staff  workload 

■  Increase  patient  control  and  empowerment 


PCMH  is  the  lynchpin  to  better  decision 
support  for  patients  and  health  teams 
General  Green,  SG  USAF 


Integrity  -  Service  -  Excellence 


Integrity  -  Service  -  Excellence 
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Project  Inception  and  Evolution 


U.S.  AIR  FORCE 


Project  Organization  Chart 


■  Initiated  in  2008  by  AFSG 

■  Joint  Base  at  Elmendorf-Richardson  (JBER)  selected 

■  Approximately  37,000  patients;  medical  group  staff  of 
approximately  150 

■  Project  Team  consisted  of  4  sub-teams  each  focused  on  specific  tasks 

■  Functional  Team 

■  Technical  Team 

■  Change  Management  Team 

■  Study  Design  Team 


Integrity  -  Service  -  Excellence 


Integrity  -  Service  -  Excellence 


u.s.  »IR  FORCE 


System  Rollout  &  Change 
Management 


U.S.  »IR  FORCE 


Patients’  Perspectives:  Benefits 


SYSTEM  ROLLOUT 

Key  activities  requiring  completion  prior  to  go-live  included: 

■  Obtaining  information  assurance  compliance 

■  Establishment  of  a  production  support  team 

■  Revisiting  business  rules 

■  User  acceptance 


CHANGE  MANAGEMENT 

Engage  stakeholders  throughout  the  process  with  primary  goals  to: 

■  Reduce  resistance  from  and  promote  adoption  by  medical  group 
staff 

■  Drive  demand  of  and  promote  adoption  by  patient  population 

■  Proactively  identify  and  mitigate  risks 

■  Prepare  medical  group  to  effectively  use  the  tool  and  integrate 
secure  messaging  into  existing  clinical  processes 


Integrity  -  Service -  Excellence 


Integrity  -  Service-  Excellence 
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Patients’  Perspectives:  Concerns 
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Branding  &  Messaging 


■  MiCare  chosen  as  the  name  of  the  PHR/SM  solution 

■  Central  marketing  theme  “Take  Command  of  Your  Healthcare” 

■  Marketing  material  developed  to  appeal  to  different  groups  of  patients 
(e.g.,  active  duty,  parents,  retirees) 

■  Key  PHR/SM  Capabilities  and  Benefits  Used: 

■  Request  your  next  appointment 

■  Request  medication  renewals 

■  Receive  your  test  and  lab  results 

■  Maintain  a  Personal  Health  Record  (PHR)  to  manage  your  health 

■  Communicate  online  with  your  healthcare  team  about  non-urgent  symptoms 

■  Avoid  unnecessary  office  visits  and  telephone  calls 

■  Request  a  copy  of  your  immunization  record 

■  Access  a  large  library  of  patient  education  materials 


Integrity  -  Service  -  Excellence 


Integrity  -  Service  -  Excellence 


U.S.AIR  FORCE 


Branding  Samples 


U.S.  AIRFORCE 


2-Pronged  Change 
Management  Approach 
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Integrity  -  Service-  Excellence 
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Patient  Enrollment  Process  & 
System  Interaction 


U.S.AIW  FORCE 


Research  Study  Overview 


m  Goal:  assess  impact  of  PHR/SM  on  patient/provider  satisfaction  and 
clinical  and  operational  outcomes;  leam  best  practices 


■  Data  collected  from  3  sources  at  baseline,  6  months  and  12  months  to 
provide  sufficient  time  for  outcomes  to  be  influenced 


Survey  data:  perceptions,  attitudes 
Medical  data:  health  outcomes 
Service  data:  resource  utilization 
Comparison  with  control  group 


Integrity  -  Service -Excelle 


Integrity  -  Service-  Excelle 
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Who  are  the  Early  Adopters? 


How  are  they  Using  the  Tool? 


Variable 

Elmendorf 

PHR/SM  Earty 

Survey 

Population  (1) 

Adopters  (2) 

Respondents  (3) 

Demographics 

Gender  (Male=1) 

« 

.37(1*) 

.36(1*) 

Age 

40.0 

32  un 

47211*2*) 

Sponsor  Pay  Grade 

5.48 

5.40(1*) 

5.69(1*, 2*) 

Number  of  Dependents 

72 

63 

60 

Dependents  vs  Active  Duty 
(Dependent  Category  =  1) 

52 

.64(1*) 

.55(2*) 

Medical  Condition 

Average  Total  Chronic  Diseases 

39 

49  in 

63(1*^) 

N 

20,069 

1,518 

283 

U.S.AIR  FORCE 


■  28%  (compared  to  25%  of  survey  sample)  of  enrolled  patients  either 
sent/received  messages  or  accessed  their  PHR  in  first  month 


Integrity  -  Service -  Excellence 


Integrity  -  Service  -  Excellence 


Study  Highlights  - 
Patient  Expectations 

■  Patients  are  overwhelmingly  positive 

■  /  think  on  line  contact  with  doctor  &  medical  records  will  answer 
lots  of  questions  and  save  so  much  time  for  both  doctor  and 
patients. 

■  I'm  excited  at  this  opportunity,  and  feel  like  it  is  always 
empowering  to  be  able  to  communicate  via  the  internet  about 
routine  health  issues  -  and  can  very  well  prevent  visits  to  the 
clinic. . 

■  /  am  really  excited  to  use  MiCare  I  was  extremely  impressed 

that  it  has  compiled  all  of  my  health  information  from  all  of  my 
providers.  Being  in  the  military  and  moving  a  lot,  sometimes  you 
forget  things  or  names  and  this  is  an  extremely  valuable  tool  to 
me.  V 

Integrity-  Service-  Excellence 


Study  Highlights  - 
What  Patients  Find  of  Value 


■  Top  5  PHR/SM  features  of  18 


Emergency  information 
immediately  accessible 

Print  and/or  exchange  medical 
information  with  providers 

Securely  message  my 
providers 

■■HBHBHBB 

Medical  information  available 
anytime  from  any  web-.. 

BHBH^B^BBHB 

Track  lab  results 

5.00  5.50  6.00  6.50 

Integrity  -  Service-  Excellence 
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Study  Highlights  - 
Change  Management  Effectiveness 


U.S.  AIR  FORCE 


Study  Highlights  - 
Provider  Expectations 


Personal  Tactics 

Registration  desk 

Provider/staff 

recommendation 


y  correlated  with: 
Use  intentions 
Value  perceptions 
Compatibility  perceptions 
Empowerment  potential 


■  Providers  express  concern  in  open-ended  comments 

■  Initial  promotion  made  many  promises  which  have  not  been  bom  out  in 
practice  and  use  It  has  increased  workload  tremendously  without  improving 
patient  satisfaction 

■  Providers,  nurses,  techs  and  patients  have  all  been  frustrated  by  what  is 
lacking. 

u  I  realize  that  MiCare  is  a  work  in  progress ,  but  the  content  on  the  patient  side 
is  not  currently  as  rich  as  I  had  been  led  to  believe  it  would  be  Many  lab 
results  seem  to  be  missing. 

■  MiCare  does  not  communicate  with  our  current  electronic  medical  records 
system-so  everything  we  do  in  MiCare  that  needs  to  be  part  of  the  official 
medical  records  causes  me  to  perform  the  tasks  twice — waste  of  time! 

-% 

. 


Integrity  -  Service  -  Excellence 


Integrity  -  Service  -  Excellence 


Study  Highlights- 
JL.  Provider/Patient  Perceptions 


U.S.  AIR  FORCE 


Summary  of  Early  Findings 


Comparison  of  Patient  and  Provider  Perceptions 


■  Positive  “expectations”  among  patients,  less  so  for  providers 

■  Adoption  is  trending  up  but  still  low 

■  Key  is  personal  promotion  and  recommendation  to  patients  by 
providers 

■  Usage  is  low 

■  Provider-initiated  secure  messaging 

■  Identification  and  designation  of  super  users 

■  Compelling  “use"  stories 

■  Overcoming  provider  resistance 

■  Realistic  messaging  and  expectation  management 

■  Sufficient  training  of  providers/staff 

■  Workflow  and  business  rules 


Integrity  -  Service -  Excellence 
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Next  Steps 


■  September:  6  months  from  end  of  baseline  period 

■  Analyze  usage  data 

■  Match  Elmendorf  sample  with  control  group 

■  Explore  effects  on  medical  outcomes 

■  Medical  conditions 

■  Diabetes 

■  Hypertension 

■  Asthma 

■  Polypharmacy 

■  High  utilizers 


Integrity  -  Service  -  Excellence 
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Effects  of  COMPASS  Workflow  Documentation  Quality  of  Family  Medicine  Physicians  using  the 
Military  Electronic  Health  Record  (AHLTA) 

Lt  Col  Charles  Motsinger,  Workflow  Division,  Office  of  the  Chief  Information  Officer,  Air  Force 
Medical  Support  Agency 

Abstract:  Electronic  medical  records  are  touted  to  be  able  to  improve  the  documentation  of  medical  care.  To 
date  there  are  no  studies  applying  a  standardized  clinical  workflow  to  an  electronic  medical  record.  AIMS:  To 
determine  if  the  COMPASS  workflow  improves  the  documentation  and  coding  of  family  physicians  using  the 
military’s  electronic  medical  record  (AHLTA).  Method:  189  charts  were  reviewed  retrospectively  from  two  Air 
Force  family  medicine  residency  sites.  Primary  outcomes  were  compliance  with  Joint  Commission  (JC)  and 
Health  Services  Inspection  (HSI)  requirements  for  outpatient  documentation,  relative  value  units  (RVU’s)  per 
encounter,  coding  accuracy,  and  readability  of  notes.  Results:  The  COMPASS  workflow  is  associated  with  a 
significant  increase  in  compliance  with  JC  and  HSI  requirements  (P<.05),  a  significant  increase  in  RVU’s  per 
encounter  (P<.05),  a  significant  increase  in  coding  accuracy  (P<.05)  and  a  significant  increase  in  readability  of 
notes  (P<.05). 
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Methods  and  Results 


Integrity  -  Service  -  Excellence 

Effects  of  COM  PASS  Workflow  on 


■  IRB  approved  research  comparing  documentation  at 
Travis  (non-COMPASS)  and  Nellis  (COMPASS). 


Documentation  Quality  of  Family  Medicine 
Physicians  using  the  Military  Bectronic 
Health  fecord  (AHLTA) 

i  A 


W 


U.S.  AIR  FORCE 


Workflow  Division 
Office  of  the  Chief  Information  Officer 
AFMSA936 
2  Aug  2011 


■  Participants:  Teaching  staff  and  Intern  records:  250  charts 

■  Results:  COMPASS  workflow  associated  with 

■  Increased  Joint  Commission/HSl  compliance 

■  Increased  Readability  of  notes 

■  Increased  coding  accuracy 

■  Increased  RVU’s  per  encounter 


Integrity  -  Service  -  Excellence 


Joint  Commission/HSl 
Compliance 


U.S.  AIR  FORCE 


Readability  of  Note 
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«•*  Coding  Accuracy 
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Non  GbmpassSaff 

49% 

Non  Cbmpass  Interns 

27% 

GbmpassSaff 

64% 

2=2289 

Compass  Interns 

73% 

2=4  666 

Ceding  aecureey  perwniege 


Integrity  -  Service  -Excellence 


w 

❖ 

S-  AIR  FORCE 

RVU’s  per  Encounter 

Non  Compass 
Staff 

Compass  Staff 

Non  Compass 

CwT 

Provider  Entered 

0.972 

1.15 

0.943 

126 

Corrected 

1.054 

1.164 

1.067 

1.152 

RVU  per  Encounter 
or  Co  "icess  Salt  eCorcen 


RWJ  pec  Encounter 

■  Mop  Cc.-rcess  Intemi  ■C3*'ksj  in 
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QUESTIONS 
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Teamwork  Factors  Affecting  Safe  Blood  Product  Administration 
Maj  Jennifer  Hatzfield,  Travis  AFB,  United  States  Air  Force 

Background:  Blood  transfusion  errors  are  a  potentially  fatal  mistake  that  can  occur  within  the  hospital  setting 
and  often  result  from  errors  in  patient  identification  at  the  bedside  just  prior  to  administration.  Transfusion 
errors  are  most  frequent  in  smaller  facilities  and  primarily  due  to  administering  a  correctly  labeled  unit  of  blood 
to  the  incorrect  patient. 

Methods:  Between  March  2009  and  August  2009,  the  simulation  center  at  David  Grant  Medical  Center  devised 
a  scenario  to  test  if  appropriate  patient  identifiers  were  verified  prior  to  administering  a  unit  of  packed  red  blood 
cells.  Thirteen  teamwork  activities  were  scored  for  sixteen  different  clinical  teams. 

Results:  Of  the  sixteen  simulations,  four  teams  (25%)  hung  the  incorrect  blood  for  the  patient  in  the  simulated 
environment.  One  teamwork  factor  (team  cross-monitors  and  gives  feedback)  was  statistically  significantly 
lower  for  groups  that  gave  the  wrong  blood  (p=0.03).  Four  other  items  suggested  differences  between  groups, 
but  were  not  statistically  significant  because  of  the  limited  sample  size.  These  factors  included  directing 
responsibility  to  individual  team  members  (p=0.13),  engaging  the  patient  in  treatment  (p=0.15),  making 
decisions  through  collective  input  (p=0.13),  and  clear  goals  articulated  from  the  leader  (p=0.1 1).  There  were  no 
differences  in  the  scores  from  the  other  teamwork  factors  (p=1.0  for  all). 

Conclusion:  The  simulation  environment  provides  a  valuable  avenue  to  practice  and  evaluate  high-risk 
activities,  such  as  blood  product  administration.  Additional  study  is  needed  to  determine  if  the  identified 
teamwork  items  are  significantly  different  in  a  larger  sample  size  and  in  other  high-risk  activities. 
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Blood  Product  Administration 

Blood  transfusion  errors  are  a  potentially  fatal 
mistake  often  from  errors  in  patient 
identification  at  the  bedside  just  prior  to  \ 
administration  (Murphy  et  al.,  2007) 

Risk  of  incompatible  red  cell  transfusion  is  1  in  1 
every  38,000  units  administered — 
higher  than  the  risk  of  developing  a 
viral  infection  (Linden,  Wagner, 

Voytovich  &  Sheehan,  2000) 

Transfusion  errors  are  most  frequent 
in  smaller  facilities  (<2,000  units/yr) 
mostly  due  to  administering  a 
correctly  labeled  unit  of  blood  to  the 
incorrect  patient  (Linden  et  at.,  2000). 
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Use  of  Simulation 

Simulation  is  an  important  tool  to: 
Evaluate  teamwork  and 
communication  skills  (Calhoun 
et  al.,  2009;  Rosen  et  al.,  2008; 
Manser,  2008) 

Improve  training  (Ward-Smith, 
2008;  Overstreet,  2008) 

Evaluate  factors  associated 
with  blood  product  administration 
(Liu,  Grundgeiger,  Sanderson, 
Jenkins  &  Leane,  2009). 


TeamSTEPPS® 

Stands  for:  Team  Strategies  and  Tools  to 
Enhance  Performance  and  Patient  Safety” 
Evidence-based,  teamwork  system  to 
improve  patient  safety 
Developed  by  the  DoD  Patient  Safety 
Program  and  the  Agency  for  Healthcare 
Research  and  Quality  (AHRQ) 

Comprised  of  four  specific  skills: 
Communication 


Leadership 
Situation  Monitoring 
_Mutual_Support- 


(AHRQ,  2010) 


Methods 

17  different  clinical  teams  were  evaluated 
between  March  2009  and  August  2009 
Efforts  part  of  scheduled  Team  STEPPS  training 
accomplished  in  the  simulation  center  by  every 
inpatient  unit  , 

Scenario  designed  to  test  if  appropriate  patient  I 
identifiers  were  verified  prior  to  administering  a  , 
unit  of  packed  red  blood  cells  (PRBCs). 
Scenarios  were  videotaped,  and  then  scored  by 
a  single  TeamSTEPPS  trained  staff  member  at 
the  Simulation  Center 

13  teamwork  factors  were  scored  for  each  team 


The  Scenario 


Patient  presented  with  signs/symptoms  of  low  \ 
Hemoglobin/Hematocrit  \ 

Nursing  staff  had  to  provide  telephone  report  to 
physician 

Provider  directed  nursing  staff  to  request  1  unit  of 
packed  red  blood  cells,  and  he/she  would  arrive  , 
shortly 

When  physician  arrived,  brought  the  unit  of  • 

PRBCs,  and  instructed  the  nursing  staff  to  give  it  t 
“right  away”  / . 

The  PRBCs  brought  by  the  provider  were  for 
different  patient  (different  name,  and  different 
SSN)  and  was  a  different  blood  type 


23 


Proceedings  of  the  2011  AFMS  Medical  Research  Symposium 

Volume  4  Healthcare  Informatics 


Scoring 

Scoring 

Each  element  scored  as:  “Needs  Improvement”, 

Leadership 

“Met  Standards”,  or  “Exceeded  Standards” 

Leader  articulates  team  goals 

Team  members  express  common  understanding 

Verbal  Communication 

of  problem  and  roles 

Use  of  SBAR  format  (Situation,  Background, 

Leader  holds  team  members  accountable  1 

Assessment,  Recommendation) 

Team  members  empowered  to  speak-up  and  , 

“Call  Out’VDirecting  responsibility  to  a  specific  person 

challenge 

Check  Back:  Closed  loop  communication 

Situation  Monitoring/Mutual  Support 

Hand-Off/Debriefs  at  transitions  in  care 

Team  cross-monitors 

Team  includes  patient/family  in  communication 

Shared  Mental  Model 

Team  shares  information/makes  decisions  through 

Effective  Feedback 

collective  input 

Results 

17  scenarios  completed 
Represented  80  staff 
members 
16  scenarios  were 
videotaped  and  scored 
(N=16) 

4  teams  (25%) 
incorrectly  administered 
the  packed  red  blood 
cells 


Teamwork  Factors 

Scores  were  compared  between  teams  that 
“passed”  and  those  that  “failed”  using  Wilcoxori 
Signed-Rank  Test  (non-parametric  test) 

Key  Finding: 

One  teamwork  factor  (team  cross-monitors  and  1 
gives  feedback)  statistically  significantly  lower  f 
for  groups  that  gave  the  wrong  blood  (p=0.03). 
Because  of  small  sample  size  (n=16),  likely  that 
other  factors  were  associated  with  scenario 


success 


/ 

/_ 
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Teamwork  Factors 

Four  other  items  suggested  differences  \ 
between  groups: 

Directing  responsibility  to  individual  team 
members  (p=0. 13) 

[higher  for  teams  that  gave  the  wrong  blood]  i 
Includes  patient/family  in  communication  (p=0.15)i 
Make  decisions  through  collective  input  (p=0.13)  ; 
Clear  goals  articulated  from  the  leader  (p=0.11). 


Teamwork  Factors 

No  other  differences  in  scores  were  observed  from  the 
remaining  teamwork  factors. 


Verbal  Communication 

Use  of  SBAR  format 
(Situation,  Background, 
Assessment, 
Recommendation) 
(p=0.67) 

Check  Back:  Closed  loop 
communication  (p=1.0) 
Hand-Off/Debriefs  at 
transitions  in  care  (p=1  0) 
Team  shares 
information/makes 
decisions  through 
-collechve-ifrput-(p=+.0)-  -  ' 


Leadership 

Team  members  express  common!, 
understanding  of  problem  and  rol^s 
(P=10)  | 

Leader  holds  team  members 
accountable  (p=1.0)  | 

Team  members  empowered  to  j 
speak-up  and  challenge  (p=1.0), 

Situation  Monitoring/  ^ 

Mutual  Support  / 

Shared  Mental  Model  (p=  1  .p) 
Effective  Feedback  (p^l.Q)' 


Implications  for  Local  Practice 

Scenario  highlighted  the  need  for  further 
process  improvement  in  blood  product 
administration 

(Blood  product  checklist  completed  in  2010)  \ 

Reinforces  the  need  for  continued  j 

TeamSTEPPS  training  using  simulations 

(Is  the  investment  of  staff  time  worthwhile?)  I 

Supports  the  need  for  units  to  evaluate  and 
improve  teamwork  factors 

(Does  a  teamwork  score  really  matter?K^  / 


Implications  for  Patient  Safety 

Specific  teamwork  factors  appear  to  be 
associated  with  poor  patient  safety 
outcomes 

Initial  and  recurring  teamwork  training 
should  emphasize  the  importance  of  each  1 
teamwork  factor 

Simulations  can  provide  a  safe  environment 
to  learn  from  failure  . 

Direct  feedback  for  the  participants 
Ability  to  evaluate  trends  within  an  organization 
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Limitations 

Limited  sample  size 
Accomplished  at  one  location 
TeamSTEPPS  teamwork  factors  have 
face  validity,  but  scores  have  not  been 
validated  with  established  teamwork 
measures 

Inter-rater  reliability  of  scores  has  been 
a  problem  in  the  past,  reason  for  a  ^ 
single  rater  in  this  project 


Future  Research 

Expand  scenarios  to  include  other  low- 
volume,  high-risk  activities:  is  success 
associated  with  the  same  teamwork  factors 
What  is  the  role  of  experience  and 
expertise  on  teamwork  and  patient  safety 
outcomes 

Explore  the  relationship  among  teamwork 
factors  (communication,  leadership, 
situation  monitoring  &  mutual  support) 
Further  validation  of  teamwork  scoring 
elements  is  needed 


“Thwi  @r@  m  i 
success.  Ht  te  tine  mm&i 
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Analysis  of  Population-Level  Data 

Dr  Ryan  Mayes,  United  States  Air  Force,  Wright-Patterson  AFB,  OH 

Sampling  techniques  and  statistical  tests  are  required  to  estimate  population  parameters  in  the  absence  of  data 
for  a  fully  enumerated  population.  However,  in  the  military,  it  is  often  the  case  that  population-level  data  are 
available.  This  raises  two  interesting  questions:  (1)  when  a  population  is  fully  enumerated,  is  it  appropriate  to 
apply  sample-based  techniques  (hypothesis  tests,  confidence  intervals,  etc.)  and  (2)  if  not,  what  procedures 
should  be  used?  This  presentation  will  address  both  questions.  Discussion  of  the  first  question  will  review  why 
it  is  inappropriate  to  simply  treat  population  data  as  a  sample  of  a  larger  population  and  use  sample-based 
testing.  Sample-based  techniques  are  not  needed  to  estimate  a  parameter  if  that  parameter  can  be  calculated;  it 
is  not  appropriate  to  apply  these  techniques  to  data  for  a  fully  enumerated  population.  The  second  question  will 
address  alternatives  to  sample-based  testing.  Hypothesis  tests  answer  the  question  of  whether  a  difference 
between  a  parameter  and  a  sample  statistic  (or  between  two  statistics)  is  likely  real  (“significant”)  but  remain 
silent  on  whether  the  difference  is  important.  When  comparing  two  parameters,  any  detected  difference  is  real 
-  a  hypothesis  test  would  be  of  no  use.  Because  differences  are  very  likely  to  occur,  determining  whether  a 
difference  is  important  becomes  the  predominant  task.  To  evaluate  the  importance  of  detected  differences, 
options  based  on  both  magnitude  and  probability  will  be  presented.  The  magnitude-based  option  sets  a  priori 
differences  in  effect  sizes,  while  the  probability-based  option  uses  a  non- sample-based  z-test  (using  the 
population  standard  deviation  rather  than  a  standard  error).  Multivariable  analyses  will  also  be  discussed. 
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Integrity  it  Service  *  Excellence 


Analysis  of 
Population-Level 
and  Large-Sample 
Data 

2  August  2011 

Ryan  Mayes,  Ph.D.,  MPH 
USAFSAM/PHR 
(Epidemiology  Consult  Service) 
Wright-Patterson  AFB,  OH 


The  Issue 


•  In  general,  statistical  tests  were  designed 
with  the  idea  of  taking  a  small  sample  from  a 
large  population. 

o  This  is  often  due  to  necessity,  either  practical  or 
financial. 

•  However,  this  is  not  always  the  case  with  the 
military,  which  has  vast  stores  of  data  on 
most  (if  not  all)  of  its  population  along  many 
dimensions. 


.^AFFtLT.  1 
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The  Issue 


•  Despite  the  difference  in  data  availability 
between  military  and  civilian  agencies,  the 
military  often  uses  the  same  data  analysis 
techniques  that  were  developed  for 
relatively  small  samples  of  large 
populations. 


•  This  may  not  be  an  appropriate  approach  - 
for  instance,  an  often-forgotten  guideline  is 
that  most  statistical  tests  of  significance  are 
designed  for  samples  no  larger  than  5%  of 
the  population  size. 


. A  RL 


The  Issue 


9 


•  Let’s  consider  a  (fabricated)  example.  Suppose  we 
want  to  know  if  the  systolic  blood  pressure  (SBP)  of 
active  duty  Air  Force  (ADAF)  Public  Health  (PH) 
personnel  is  substantially  different  from  ADAF 
personnel  in  general. 

•  An  Armed  Forces  Health  Longitudinal  Technology 
Application  query  yields  the  following* *: 

O  //(mean  SBP  of  all  ADAF)  =  115.0 
oX  (mean  SBP  of  1000  PH  personnel)  =  115.5 
o  s  (standard  deviation  of  SBP  for  1000  PH)  =  4.5 
o  <7  (standard  deviation  of  SBP  for  all  ADAF)  =  4.5 
o  n  (number  of  PH  “sampled”)  =  1000 

'Data  are  fabricated  and  are  entrely  for  illustrative  purposes.  _ 
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The  Issue 


•  With  these  data  for  1000  individuals,  there  are 
three  possibilities  for  the  relationship  between 
sample  and  sample  population: 

1.  The  data  could  be  a  small  sample  (<5%)  of  a  large 
population  (this  would  mean  that  the  entire  population 
of  PH  personnel  would  need  to  be  20,000+). 

2.  The  data  could  be  a  large  sample  relative  to  the 
population  size  (such  as  1000  sampled  out  of  2000 
total). 

3.  The  data  could  comprise  a  census  (an  entire 
population). 

•  We’ll  examine  approaches  for  each  of  these 
possibilities. 


Scenario  1:  Small  Sample 


•  Generally,  we  would  approach  this  problem  as  a 
typical,  basic  hypothesis  test.  This  is  entirely 
appropriate  for  a  relatively  small  sample.  We’d  set  up 
our  null  hypothesis  as  something  like  H0:  ju  =  120. 

•  We  would  next  run  the  test: 

(y-g)  (115.5-115.0) 

(s)4»  4.5/ Vl  000 

l - - - *  or  a 

•  This  yields  a  very  small  p-value  (0.000442);  we 
conclude  that  the  SBP  of  PH  personnel  is 
significantly  different  from  that  of  the  AF  in  general. 

a»iiM»  ww-a-mo.  'U«A  1 


Scenario  2:  Large  Sample 


•  If  the  sample  of  n=1000  came  from  a  population 
of  N=2000,  it  would  no  longer  be  appropriate  to 
use  a  typical  hypothesis  test. 


o  One  of  the  underlying  assumptions  of  sampling 
theory  is  violated;  our  sample  is  well  above  5%  of 
the  population  size*. 


The  relatively  large  sample  size  would  prompt  the 
use  of  the  finite  population  correction  (FPC): 


<rT  = 


ThisistheFPC  which 
is  added  to  the  usual 
star-na*  a  aror  term 


jn'Xn- 1 

o  Where  y  is  the  populatiorfslze  and  n  is  the  sample 
size. 


Scenario  2:  Large  Sample 


If  the  sample  of  n=1 000  came  from  a  population  of  N=2000, 
the  relatively  large  sample  size  would  prompt  the  use  of 
the  FPC: 


(115.5-115.0) 


a*  o  ( 

j N-n  ) 

4.5  | 

f  12000-1000  1 

M 

V  N-l  ) 

VI 000  1 

2000-1  j 

=  4.968 


Here,  p=0.000000339. 


o  Recall  that  z=3.513  and  p=  0.000442  using  traditional 
analysis. 


This  is  good  news:  compared  to  the  results  using 
traditional  analysis,  analysis  with  the  FPC  produces 
stronger  evidence  (and  has  the  benefit  of  being 
technically  correct). 


*  Cochran  WG.  Sampling  Techniques.  S^ed  New  York:  John  Wiley  &  Sons.  t©77 
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Scenario  2:  Large  Sample 


•  Take-home  message:  when  the  sample  is  a 
large  proportion  of  the  population  (>5%), 

use  the  FPC* *!  _ 

cr  <j  \N -n 

o  Substitute  =  with  — —  J - 

^n\N- 1 


•  This  appropriately  adjusts  for  the  relative 
sample  size  and  provides  stronger  evidence 
for  hypothesis  tests. 


*  CocJiran  WG  SampJing  Techniques  3"  ad.  Nev#  York:  John  Wiley  &  Sons.  1677. 


Scenario  2:  Large  Sample 


•  The  FPC  can  also  be  used  with  confidence 
intervals  (CIs): 


95%  Cl  = 


x± 


a  \N-n 
j4n\N-l 


•  This  will  make  the  confidence  interval 
narrower  (more  precise)  as  compared  to  an 
interval  without  the  FPC. 


.AFRC  10 


Scenario  3:  Census 


*  Before  delving  in  to  census  analysis,  let’s  review  the 
effect  of  sample  size  on  hypothesis  tests. 


o  Imagine  that  we  had  taken  a  smaller  sample  (n=100)  of 
PH  personnel:  .  v  . 

-_fr-/')-(“5.0-115.5)_222; 

s/Jn  4.5  /%!l00 

>This  yields  a  p-value  of  0.0263.  We  would  still  reject  the  null, 
but  the  evidence  isn’t  as  strong  .  .  . 


o  If  we  take  a  still  smaller  sample  (n=50), 

_  (*-;,)  (115.5-115.0) 

s/jn  4.5/^150 

>Here,  p=0.116:  we  would  conclude  that  the  SBP  of  PH 
personnel  does  not  differ  significantly  from  all  ADAF. 


Scenario  3:  Census 


❖ 

•  It’s  no  mystery  that  a  larger  sample  size  generally 
results  in  smaller  p-value,  but  is  this  always 
appropriate? 


•  Consider  the  case  where  the  “sample”  of  n=1000  is  not 
actually  a  sample  but  rather  a  census  of  all  PH 
personnel  (i.e.,  the  population  is  1000,  all  of  which 
were  included  in  the  analysis). 


•  In  this  case,  we’re  no  longer  dealing  with  the  sampling 
distribution  of  sample  means  but  rather  with  two  fully 
described  populations. 


•  Is  hypothesis  testing  appropriate  in  this  situation? 


„AFRL  12 
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Scenario  3:  Census 


•  If  we  are  dealing  with  a  population  rather  than  a 
sample,  the  typical  z-test  is  no  longer  appropriate, 
since  we’re  not  analyzing  a  sample: 


•  Instead,  we  would  calculate  a  z  score  for  the 
population: 

_  (x-ft)  (115.5-115.0) 
cr  4.5 

•  Here,  p= 0.824  but  has  a  very  different  interpretation 
than  the  previously  calculated  p-values. 


o  In  general,  it  means  that  there  is  an  82.4%  chance  that  a 
given  individual  from  this  population  will  produce  an 
average  SBP  at  least  this  high. 


Scenario  3:  Census 


•  There  are  several  approaches  one  could  take 
when  population-level  data  are  available: 

o  Treat  the  data  as  a  sample  from  a  super-population. 
oTake  a  sample  of  the  population-level  data, 
o  Examine  the  magnitude  of  difference. 


,sjuAFRL 
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Scenario  3:  Census 
Super-Population 


•  Perhaps  the  simplest  approach  is  to  treat  the 
population  as  part  of  a  larger,  hypothetical  super¬ 
population. 


o  In  this  case,  one  would  proceed  using  normal  hypothesis 
testing  methods  (remember  to  use  the  FPC  if  the  population  is 
anticipated  to  be  larger  than  5%  of  the  super-population). 


o  This  is  often  the  approach  used  in  existing  literature. 


•  This  may  or  may  not  be  appropriate.  As  a  guideline, 
if  the  problem  at  hand  is  specific  to  a  particular  group 
(place,  time,  etc.),  the  super-population  approach 
probably  doesn’t  apply. 


o  This  applies  to  many  AF  queries  -  the  group  of  interest  often 
consists  of  current  personnel,  not  necessarily  future/unknown 
personnel. 


Scenario  3:  Census 
Subsample 


•  Another  approach  is  to  subsample  the 
population  for  which  you  have  full  data. 


fi 


oOne  advantage  is  that  traditional  hypothesis  tests 
would  be  appropriate  for  analysis. 


•  However,  this  has  some  theoretical  issues: 
the  purpose  of  hypothesis  testing  is  to 
estimate  an  unknown  population.  We  don’t 
actually  need  to  do  that  if  we  have  data  for  the 
entire  population. 


o  Therefore,  this  approach  is  only  recommended  if 
formal  hypothesis  testing  is  absolutely  required 
(i.e.,  for  publication). 
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Scenario  3:  Census 
Magnitude  of  Difference 


•  Hypothesis  tests  answer  whether  a 
difference  is  statistically  significant  -  in 
other  words,  whether  a  difference  is  “real.” 

•When  dealing  with  population  data,  this 
question  is  irrelevant  -  any  difference 
between  two  populations  is  a  real  difference. 

•What  should  be  addressed  is  whether  a 
difference  is  important. 
o  Statistical  significance  vs.  clinical  significance 


Scenario  3:  Census 
Magnitude  of  Difference 


•  In  the  SBP  example,  there  is  a  demonstrable 
difference  between  the  ADAF  and  PH  SBPs. 
As  shown  above,  this  difference  is 
statistically  significant. 

•  However,  the  difference  is  0.5  (115  vs.  115.5), 
which  is  not  a  large  clinical  difference. 
oEven  though  the  difference  is  real,  is  it  important? 
o  Should  any  action  be  taken  based  on  this 

disparity? 


_ 
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Scenario  3:  Census 
»;*  Magnitude  of  Difference 

Scenario  3:  Census 
-•;*  Magnitude  of  Difference 

1 

•  A  thorough  literature  review  has  not  found  any 
established  methods  of  analyzing  population-level 
data. 

•  Therefore,  it  is  recommended  that  the  magnitude 
of  the  difference  between  groups  be  considered. 

•  The  simplest  way  to  do  this  is  to  look  at  the 
absolute  difference;  an  a  priori  benchmark  must 
be  set. 

o  If  the  PH  SBP  is  at  least  5%  higher  than  the  ADAF 

SBP,  the  difference  will  be  considered  important. 
oHere,  the  PH  SBP  is  only  about  0.4%  higher,  which  is 
not  a  large  difference. 

•  A  slightly  more  sophisticated  approach  (and 
the  recommended  one)  would  be  to  base  the 
magnitude  of  difference  on  the  population 
standard  deviation  cr. 

o  Again,  an  a  priori  limit  will  need  to  be  set,  this  time 
in  terms  of  a  proportion  of  a. 

•  Example:  a  difference  will  be  deemed 
important  if  it  exceeds  20%  of  a. 
oHere,  a  =4.5.  4.5*(0.20)  =  0.9. 

oThe  difference  is  (115.5-  115.0)  =  0.5,  which  is  less 
than  the  a  priori  standard.  Therefore,  we  would  not 
consider  this  difference  to  be  important. 

» _ _.*..«,*** _ _ 20 
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Scenario  3:  Census 
Magnitude  of  Difference 


•  Formally: 

5  =  (x-ju)-(cjy) 


o  Where  y  is  the  desired  proportion  (20%  in  the 
previous  example)  to  be  taken  from  a. 
o  If  S>0,  the  difference  is  considered  important. 

•  Compare  this  to  a  z-test 

X-f.1 

z  —  1 

o  Note  that  6  is  a  difference  statistic  ,  while  z  is  a 
proportion. 
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Scenario  3:  Census 
Magnitude  of  Difference 


•  Advantages 


□  Appropriate  for  population-level  data 
o  Easy  to  apply 

o  Flexible:  to  be  more  conservative,  use  a  smaller 
proportion  of  a 

□  Can  be  used  with  multiple  data  types  (continuous, 
discrete,  etc.) 


•  Disadvantages 

o  Not  in  common  use,  may  still  need  to  supplement  with 
a  p-value 

o  Fairly  subjective,  no  standard  proportion  of  a  in  place 

>^0f  course,  a  is  also  highly  arbitrary 


Multivariable  Analyses 


•  For  multivariable  analyses  (regressions, 
survival  analysis,  etc.)  of  large  samples,  the 
FPC  can  be  used. 

o  It’s  possible  to  do  this  manually,  but  this  approach 
would  be  challenging. 

o  Use  of  the  FPC  for  multivariable  analyses  is 
automated  in  many  statistical  software  packages. 
It  is  generally  accessed  through  the  sampling 
options. 

>Note,  however,  that  this  may  result  in  (too)  many 
variables  being  significant. 


Multivariable  Analyses 


•When  dealing  with  full  population  data,  the 
super-population  approach  is  very  tempting. 


o  This  approach  may  be  more  appropriate  for 
multivariable  analyses.  Instead  of  whether  there 
is  a  significant  difference,  multivariable  analyses 
usually  seek  to  answer  which  variables  are 
important. 

o  This  information  could  be  germane  to  future 
members  of  the  population. 


•  Future  research  will  focus  on 


developing/finding  alternative  methods. 


.AFRL:  23 


24 


33 


Proceedings  of  the  2011  AFMS  Medical  Research  Symposium 

Volume  4  Healthcare  Informatics 


W 


Appendix 


•  FPC  for  Proportion  of  Finite  Populations*: 

\pQ-p)  \N-n 


f 


\  N -1 


1  Qochran  WG  ampting  Techniques  3m  ed  New  \brie  Jbhn  Wiley  &  Sbrs  1 977. 
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Sampling  Distribution  of 
Sample  Means 

o  For  a  given  population,  the 
sampling  distribution  is  the 
distribution  of  all  possible 
sample  means  for  a  particular 
sample  size  n. 
o  The  mean  of  the  sampling 
distribution  is  equal  to  the 
population  mean, 
o  The  standard  deviation  of  the 
sampling  distribution  is  smaller 
than  the  population  standard 
deviation  u.  How  much  smaller 
depends  on  the  sample  size. 


:,AFRL 


Appendix 


•  Why  does  the  FPC  only  apply  above  5%  of 


the  sample  size? 

•  Given  A£=1000,  the  value  of  the  FPC  with  3%  of  the 
population  (n=30)  is  very  close  to  1: 

\N-n  ,'l  0.000 -50 

- =  — - =  0.998 

\N- 1  \  10,000-1 

•  However,  as  the  proportion  of  the  population 
sampled  increases,  the  value  of  the  FPC  becomes 
much  lower.  At  20%  of  the  population  (n=200): 

|.V-n  _  Iiooo6=3o6o_ 

ViV-1  10;000-1 
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The  Research  Maze  and  the  Wheels  of  Progress:  How  the  Health  Services  Data  Warehouse  Will 
Transform  the  Way  Research  is  Done. 

Lt  Col  David  Carnahan,  Office  of  the  Chief  Information  Officer,  Air  Force  Medical  Support  Agency 

There  are  many  challenges  inherent  in  every  observational  study.  In  the  Military  Healthcare  System  (MHS),  one 
of  the  biggest  is  accessing  the  many  TB’s  of  data  that  represent  the  medical  care  of  9.6  million  MHS 
beneficiaries.  The  current  process  requires  a  clinical  analyst  to  access  multiple  data  sources  with  non- 
normalized  files,  determine  relationships  between  the  files,  write  computer  code  to  establish  linkages,  which 
ultimately  transforms  flat  files  into  analytic  datasets  needed  for  analysis.  In  some  cases,  to  answer  the  research 
question  appropriately  requires  multiple  individuals  to  bring  together  data  across  different  organizations  to 
develop  the  dataset.  This  can  be  a  great  source  of  frustration,  and  a  great  deal  of  time  spent  which  creates 
inertia,  and  hinders  important  health  services  research.  By  using  a  data  warehouse,  the  data  has  already  been 
brought  together  into  a  single  source  for  researchers,  saving  time  and  effort  in  completing  research  projects, 
which  allows  a  greater  amount  of  projects  to  be  accomplished.  We  will  demonstrate  efficiencies  gained  using  a 
data  warehouse  to  source  the  data  by  comparing  it  to  current  MHS  practice  of  data  acquisition  and  analysis 
using  non-normalized  data  sources.  The  warehouse  that  has  been  created  is  named  the  Health  Services  Data 
Warehouse.  We  will  be  accessing  the  data  warehouse  using  a  data  mart  via  SAS  Enterprise  Business 
Intelligence  for  analysis.  To  demonstrate  practical  application,  we  will  use  a  research  question  on  Traumatic 
Brain  Injury  and  Mental  Health  as  our  proof  of  concept. 
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Tjf  Introduction 

U.S.  AIR  FORCE 

Integrity  -  Service-  Excellence 

The  research  maze  and  the 
wheels  of  progress 

KJ 

Lt  Col  David  H  Carnahan,  MD  MSCE 
AFMSA,  Healthcare  Informatics 

Jim  Leonard,  MPH 

U.S.  AIR  FORCE  Intellicog  Solutions 

■  Aims 

■  Background 

■  Hypotheses 

■  Methods 

■  Results 

■  Conclusions 

Integrity  -  Service  -  Excellence 

Aims 

U.S.  AIR  FORCE 

\f  The  Problem 

U.S.  AIR  FORCE 

■  To  determine  efficiency  gained  by  restructuring  current  data 
sources  into  an  organized  (normalized)  data  warehouse 

■  To  determine  the  validity  of  the  current  Health  Services  Data 
Warehouse  architecture 

“Information  is  a  source  of  learning.  But 
unless  it  is  organized,  processed,  and 
available  to  the  right  people  in  a  format 
for  decision  making,  it  is  a  burden,  not  a 
benefit.’’ 

-  William  Pollard 

Integrity  -  Service  -Excellence 

Integrity  -  Service  -  Excellence 
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Secondary  Data 
Research  Process 


U.S.AIR  FORCE 


The  Data  Sources 


m  Develop  research  question 

■  Establish  data  source 

■  Learn  data  elements  for  study  design/IRB 

■  Get  approval  to  use  data  (Data  Use  Agreement) 

■  IRB  approval 

■  Data  acquisition  (flat  files) 

■  Data  quality  checks/validation 

■  Data  manipulation  (flat  files  dataset) 

■  Statistical  analysis 

■  Manuscript 

■  Publication 


■  Data  Aggregated  from 
over  400  sites 

■  Over  140  different  data 
feeds 

■  25  Tb  of  data 

■  65  Million  records  daily 

■  100  different  daily  or 
monthly  data  products 

■  All  core  health  service 
lines  available— most 
MHS-wide 


iJBSBOHBflflfl  SQSBB 


!  HSS 
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Rules  of  Normalization 


i  Rule  #1  -  Eliminate  Repeating  Groups 
Rule  #2  -  Eliminate  Redundant  Information 
i  Rule  #3  -  All  Entities  Must  Depend  on  their  Primary  Keys 
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Current  state  of  ‘normalization’ 


Network 

Inpt 

1  Network 

1  Outpt 

1  Direct  Care  | 

■  mpt  1 

1  Direct  Care  ^ 

|  Outpt  1 

personid 

personid 

pabent_unique_id 

[universal  pabent  id] 

Patient_name 

Patient_name 

Pabent_dob 

Patient_dob 

Dateofbirth 

DateOfBirth 

Pabent_gender 

Pabent_gender 

Gender 

Gender 

1  primary  dx 

1  pnmary  dx 

8  diagnoses 

4  diagnoses 

8secdx 

4secdx 

1  primary 

1  procedure 

8  procedures 

4  procedures 

procedure 

code 

5  sec  procedure 
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Sources: 

-  Redundant 

-  Non-Norma) 
Limited  History 


a 

cm 


Transformational  Approach 


H  SDW  Core: 
Non-Redundant 
3™  Normal  Form 
Retains  History 


nW 

□ 


MHDM  (and  other  data  marts): 

Business  Intelligence  Oriented 
-  Managed  Non-Normal  Form 
(Star  Schema) 

-  Retains  History 
(Type  2  Changing  Dimensions) 

Diagnoses 
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tjf  Design  Philosophy 

U.S.  AIR  FORCE 

«•*  TBI  Demonstration  Study 

U.S.  AIR  FORCE 

■  Eliminate  redundancy 

■  Allows  matching  of  like  variables 

■  Renames  variables  in  consumer  friendly  terms 

■  Patient-centered 

■  Consolidation 

■  Retain  history 

■  To  test  our  aims,  we  used  a  demonstration  study  to  compare  the 
efficiency  and  validity  of  the  HSDW  PCMH  Data  Mart  versus  the 
current  non-normalized  data  sources  from  AFCHIPS 

■  The  study  chosen  was  Traumatic  Brain  Injury  (TBI)  and  the 
prevalence  of  selected  mental  health  outcomes  (Affective 
disorders  and  Anxiety  disorders)  -  a  well  established  association 
in  the  literature 

Integrity  -  Service  -  Excellence 

Integrity  -  Service  -  Excellence 
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TBI  and  Mental  Health  Study 


US.OIRFOWCE 


Hypotheses 


u  Traumatic  brain  injury  is  damage  to  the  brain  resulting  from 
external  impacts  from  rapid  deceleration,  impacts,  blast  waves.* 1 

■  All  traumatic  brain  injuries  are  head  injuries.  TBI  is  usually 
classified  based  on  severity,  anatomical  features  of  the  injury,  and 
the  mechanism  (the  causative  forces).2 

■  TBI  may  cause  emotional  or  behavioral  problems  and  changes  in 
personality.3  These  may  include  emotional  instability,  depression, 
anxiety,  hypomania,  mania,  apathy,  irritability,  and  anger.4 

i  MaasAi,  stocchettl  N.  Bullock  R  (August  2008)  Lancet  Neurology  7  (8):  728-41. 

i  Saatman  KE.Duhalme  AC  era)  Workshop  scientific  Team  and  Advisory  Panel  Memoera  (2008).  journal  or 

Neuroirauma25  (7):  715-38. 

I.  Zink  BJ  (March  2001).  Annals  or  Emergency  Ueoicine  37  (3):  318-32. 

♦  ArtingtiauB  KA,  snoalti  AM.  Price  TRP  (2005)  raxEOooir  or  Traumaac  Brain  injury.  Washington.  DC:  American 
Psychiatric  Association  pp. 55-62. 


■  The  HSDW  will  require  less  processing  time,  and  less  overall  time 
to  achieve  an  analysis  dataset 


■  The  HSDW  cohort  will  be  a  subset  of  the  AFCHIPS  cohort 

■  Due  to  the  nature  of  claims  data,  it  may  take  anywhere  from  6 
months  to  12  months  fora  claim  to  enter  the  source  data  file 

■  Because  the  HSDW  cohort  was  frozen  to  new  data  entry  since 
01  Jan  2010,  there  will  be  late  claims  captured  in  the  AFCHIPS 
database  that  have  not  made  it  into  the  HSDW 


Integrity  -  Service  -  Excellence 


Integrity  -  S e rv i c e  -  E x c ell e n c e 


w 

«•»  Methods 

U.S.  AIR  FORCE 

Bareli  Injury  Diagnosis  Matrix 

U.S.  AIR  FORCE 

■  Retrospective  cohort  design 

■  TBI  Grouper  classifies  head  injuries  as: 

■  Type  1  TBI  if  there  is  recorded  evidence  of  an  intracranial  injury 

■  Population 

or  a  moderate  or  a  prolonged  loss  of  consciousness  (LOC), 

■  Traumatic  Brain  Injury  (TBI) 

Shaken  Infant  Syndrome  (SIS),  or  injuries  to  the  optic  nerve 
pathways. 

■  From  01  Oct  2008  to  31  Mar  2009 

■  Type  2  TBI  includes  injuries  with  no  recorded  evidence  of 

■  Predictor  variable  -  TBI  Grouper 

intracranial  injury,  ana  LOC  of  less  than  one  hour,  or  LOC  of 
unknown  duration,  or  unspecified  level  of  consciousness. 

■  Covariates  -  Age,  Gender 

■  Type  3  TBI  includes  patients  with  no  evidence  of  intracranial 

injury  and  no  LOC. 

■  Outcome  variable 

■  Mental  Health  (MH)  Grouper 

■  Each  TBI  type  is  also  subcategorized  into: 

■  Coded  Behavioral  Health  ICD9-CM  tool  was  used 

■  Fracture 

■  Internal 

■  Data  sources 

■  Nerves  4 

■  AFCHIPS  (Network  Inpt/Outpt,  Direct  Care  Inpt/Outpt) 

■  HSDW  PCMH  Data  Mart 

A.Tfie  Bareli  injury  Diagnosis  Matrix,  Claaairicatlon  By  Body  Region  ana  Nature  ottfie  injury.  2005 
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V  TBI  and  MH  Classification 
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V 

❖ 
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■  Cohort  TBI  Classification  Predictor  Variable  Levels 

■  Type  1  fracture -T1  FX 

■  Type  1  internal  -  T1  INT 

■  Type  1  nervous  -  T1  NV 

■  Type  2  Fracture  -  T2  FX 

■  Type  2  internal  -  T2  INT 

■  Type  3  fracture  -  T3  FX 

■  Mental  Health  Outcome  Variable  Levels 

■  Anxiety  -  ANX 

■  Depression  -  AFD 

■  Both  -  AFDANX 

■  Unknown  -  UNK 

RESULTS 

Integrity  -  Service  -  Excellence 
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AFCHIPS  coding  effort 


MH 

Filter 


Grouper 

13  pages  of  code 
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HSDW  coding  effort 
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Diagnosis 

Description 


Diagnosis 

Description 
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Time  Comparisons 


Computer  Runtimes 

AFCHIPS 

HSDW 

Cohort  Identification 

5  hrs  53  min 

■■ 

2  hrs  22  min 

Incorporating  Cohort  Care 
Analysis 

Total 

1  hr  3  min 

2  min 

6  hrs  58  min 

2  hrs  41  min 

2  min 

5  hrs  5  min 

Analyst  Time 

AFCHIPS 

HSDW 

Pages  of  Code  Written 

Estimated  Total  Analyst  Time 

111  pages 

3  days 

6  pages 

1  day 

Integrity  - 

Service  -  Excell 

en  c  e 

w 

U.S.AIH  FORCE 

Demographics 

J  f**  Variable 

AFCHIPS 

HSDW 

Sample  Size 

27.724 

22,983 

Age 

39.7  (+26) 

38.8  (+26) 

Gender 

Female 

10,418(38) 

8,778  (38) 

Male 

17.306(62) 

14,205  (62) 

Integrity  -  Service  - 

Excellence 

TBI  Classification  Results 


U.S.  AIR  FORCE 


Mental  Health  Outcomes 


70 


T1  FX  T1  INT  T1  NV  T2  FX  T2 INT  T3  FX 

TBI  Categories 


Anxiety  Depression  Both  Unknown 

Mental  Health  Outcome 
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76/  Injury  by  Age  Groups 
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76/  categories  by  Age  group 


AF 

DW 

AF 

DW 

AF 

DW 

AF 

DW 

AF 

DW 

AF 

DW 

Age 

Group 

TI  FX 

TI  INT 

TI  NV 

12  FX 

T2INT 

13  FX 

<18 

196 

162 

1289 

1103 

11 

11 

235 

180 

3916 

3581 

41 

33 

18-30 

276 

208 

1945 

1561 

13 

12 

218 

164 

6168 

4859 

62 

44 

31-40 

54 

39 

706 

615 

3 

3 

30 

25 

1626 

1400 

16 

15 
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56 

52 
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12 

11 

45 

42 
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8 
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87 
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MH  Outcomes  by  Age  Groups 
AFCHIPS 


31-40  41-50 

Age  Groups 


51-64  65  and 

older 


aAFD 

■ANX 

BOTH 


Unknown  Group 
Excluded  from 
Graph 
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MH  Outcomes  by  Age  Groups 

HSDW 
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MH  Outcomes  by  TBI  Category 

AFCHIPS 
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TBI  Injury  Type 
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MH  Outcomes  by  TBI  Category 
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HSDW 
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Logistic  Regression 
for  Mental  Health  Outcomes 
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Variables 

AFCHIPS 

Odds  Ratio  (Cl) 

HSDW 

Odds  Ratio  (Cl) 

Gender 

0.80  (0  75-0.86)* 

0.80  (0.75  -0.86)* 

Age  Group 

<  18  Comparison  Group 

18-30 

9.1  (8.1-10.4)* 

8.5  (7.4-9  7)* 

31-40 

14.1  (12.3  — 16.2)* 

13.0(11.1  -15.0)* 

41-60 

10.4(8.9-12.1)* 

9.8  (8.3-11.5)* 

51-64 

6.8  (5.8-7  8)* 

6.3  (5.3-7  3)* 

>=  65 

4.7(41  -5  3)* 

4  3  (3.8  —  5  0)* 

TBI  Category 

T3  FX  Comparison  Group 

T1  FX 

1-43(0.96-2.14) 

1.86  (1  17-2.97)* 

T1  INT 

2.26(1.57-3.26)* 

2.79(1.81  -4.30)* 

T1  NV 

1.32  (0.64-2.74) 

1.39  (0.62-3.11) 

T2FX 

1.31  (0.87-1.97) 

1.55  (0.96-2.50) 

T2INT 

1.66  (1.16-2.39)* 

1  98  (1.29-3.05)* 
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Study  Limitations 


■  Data  Sources  were  updated  more  recently  in  AFCHIPS  than  in 
HSDW  which  accounts  for  the  differences  in  study  sample  size 

■  Diagnoses  in  encounter  and  claims  data  is  difficult  to  verify 
without  a  sample  chart  review  to  determine  accuracy 

■  Infrastructure  issues  could  affect  runtimes  depending  on  the  time 
of  day  the  code  was  run 


Integrity  -  Service  -  Excellence 


v  y 

«•*  Future  of  DoD/MHS  Research 


SAS  Application 


Integrity  -  Service -  Excellence 


u.s.  AIRFORCE 


Conclusions 


■  The  Health  Service  Data  Warehouse  has  demonstrated 

■  Accuracy:  Over  99.5%  of  the  population  identified  in  the 
current  population  was  found  in  the  HSDW 

■  Efficiency: 

■  Query  run  times  were  reduced  by  30%  (2  hours) 

■  Number  of  pages  the  data  analyst  needed  to  write  was 
reduced  by  95%  (from  111  pages  to  6  pages) 

■  Estimated  total  analyst  time  was  reduced  by  67%  (from  3 
days  to  1  day) 

■  The  efficiency  gains  will  be  increased  with  more  complicated 
analyses 

■  The  capacity  to  support  future  research  will  be  amplified  using 
the  HSDW  as  the  foundation  of  the  informatics  infrastructure 


U.S.  AIR  FORCE 


QUESTIONS? 
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The  Health  Services  Data  Warehouse  (HSDW)  in  Action:  Focus  on  Patient  Centered  Medical  Home 
(PCMH) 

Maj  Claudine  Ward,  Office  of  the  Chief  Information  Officer,  Air  Force  Medical  Support  Agency 

Background:  Acquisition  of  healthcare  data,  streamlined  data  management,  and  effective  information  delivery 
are  trouble  areas  within  our  current  military  medical  system.  Healthcare  data  is  collected  across  multiple 
forums,  resulting  in  confusion  among  users  and  differing  metrics  for  similar  measures  depending  on  which 
source  is  referenced.  Information  delivery  is  often  slow  and  inefficient,  resulting  in  decision  delays. 

Intro:  The  Air  Force  Medical  Service  Office  of  the  Chief  Information  Officer  (AFMS  OCIO)  is  striving 
towards  consolidating  healthcare  data  into  one  location,  the  Health  Services  Data  Warehouse  (HSDW),  to  allow 
for  centralized  data  management  and  standardized  data  transformation.  The  HSDW  will  also  focus  on 
improved  information  delivery  by  providing  easily  accessible,  usable  information  to  senior  level  leadership  and 
medical  staff  through  deliverables  such  as  push-reports  and  dashboards. 

Methods:  To  illustrate  how  the  HSDW  will  be  used,  Patient  Centered  Medical  Home  (PCMH)  data  will  be 
examined.  Rather  than  measuring  performance  measures  based  on  production,  PCMH  focuses  on  healthcare 
outcomes  of  patients  and  efficient  use  of  medical  services.  These  new  metrics  must  first  be  defined.  Once 
defined,  a  PCMH  data  mart  is  created  from  HSDW,  and  contains  the  designated  data  elements  related 
specifically  to  PCMH.  With  the  data  mart  in  place,  PCMH  push  reports  and  dashboards  are  now  created. 

Conclusion:  The  HSDW  is  intended  to  serve  as  the  ‘cornerstone  of  an  informatics  strategy  to  better  support 
clinical  decision  support,  business  intelligence,  agile  development,  and  improved  analysis  including  a  de- 
identified  research  view  of  the  data’  (FY10  Air  Force  HSDW  SOW  v4)  as  is  demonstrated  through  PCMH. 
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Headquarters  U.S.  Air  Force 


Overview 


Patient  Centered 
Medical  Home 
Metrics  and 
the  HSDW 


Claudine  Ward 
Maj,  USAF,  MC 
AFMSA / SG6 
3  August  2011 


Objectives 

Discuss  the  primary  aspects  of  PCMH 
Create  relevant,  measurable,  actionable  outcome  metrics  for  PCMH 
Share  ideas  regarding  how  to  improve  these  metrics  using  the  HSDW 


■  Patient  Centered  Medical  Home  (PCMH)  Model 


■  Current  PCMH  Metrics 


■  Wish  List 


■  Way  Ahead:  Health  Services  Data  Warehouse  (HSDW) 


■  Future  PCMH  Metrics 


Integrity  -  Service-  Excellence 
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Patient  Centered 
Medical  Home  Model 


■  Core  Features: 

■  Patient  Centered 

■  Whole  Person  Orientation 

■  Team-based  Care 

■  Coordinated  &  Integrated  Care 

■  Quality  and  Safety 

■  Enhanced  Access 

■  Payment  Reform 


Integrity  -  Service  -  Excellence 
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How  can  we  use  data 
to  evaluate 
how  we’re  doing? 


Integrity  -  Service-  Excellence 
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Current  PCMH  Measures 


Continuity  of  Care 

Emergency  and  Urgent  Care  Utilization 
HEDIS  Measures 
Patient  Satisfaction 
Specialty  Care  Utilization 
Procedure  Capture 
RVU  Business  Plan  Execution 
Support  Staff  Per  Provider  Ratio  Trend 
MTF  Weekly  Average  Appointments  Per  Provider 


Continuity  of  Care 

Emergency  and  Urgent  Care  Utilization 

Incentivized 

HEDIS  Measures 

Metrics 

Patient  Satisfaction 

Specialty  Care  Utilization 
Procedure  Capture 
RVU  Business  Plan  Execution 
Support  Staff  Per  Provider  Ratio  Trend 
MTF  Weekly  Average  Appointments  Per  Provider 
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Current  PCMH  Metrics 
Continuity  of  Care 


■  Numerator:  Patient  visits  by  PCM,  PCMH  Team,  or 
other  provider 

■  Denominator:  total  PCMH  visits 

■  Goal  is  >90%  visits  with  PCM  and  PCMH  team 


Current  PCMH  Metrics 

Emergency  Dept  and  Urgent  Care  Utilization 


■  Numerator:  ED  and  UC  visits 

■  Denominator:  100  PCMH  patients 

■  Goal  is  <3  visits  per  100  PCMH  patients 

■  May  be  incomplete  due  to  data  lag  (includes  purchased  care) 


Integrity  -  Service  -  Excellence 
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Current  PCMH  Metrics 
HEDIS  Measures 


Current  PCMH  Metrics 
Patient  Satisfaction 


■  8  HEDIS  measures  (maximum  composite  score  of  40) 

■  Scale  of  0-5  (10%,  25%,  50%,  75%,  90%) 

■  Score  based  upon  average  national  percentile  ranking 


■  Service  Delivery  Assessment  (telephone  survey) 

■  10  Questions  within  1  month  of  visit 


■  Goal  is  >  95%  Satisfaction  (score  of  4  or  5) 


Integrity  -  Service  -  Excellence 
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In 


tegrity-  Service  -  Excellence 


10 


Wish  List  for  the  Future 


Accessible  data  - 

Drill-down  and  slice  ^ 


.Flexibility  ^S®' 

■  Reprodl^^ie  Results 

■  Improved  Presentation 


Integrity  -  Service  -Excellence 


Health  Services  Data 
Warehouse 


Phase  1  -  Data  Consolidation 


m, 

X 


m 


Consoioated  Repo&cor y 


DATADEV 

^DATAPRQD^ 

COHORT 


•  Integration 
•  Transformation 


Phase  2  -  Data  Warehouse  Development 
/■ 

Hcato  S«rv,cei  Data 

*GoW  .  Reporting 
Standard" 

Source 

•Data Quality  Hietonca( 

•  Data  acc esB  Detail 

•  Conformed  .  oris  Down- 
Dimeneiona  Tnrougn-Acroaa 


C  usTooier  Produi.1* 


*k* 


f  A  PUS  Dau  Marts 

I  PC  MM 

vm 

J  HIS 

s 
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HSDW 

Transformational  Approach 


Transformational  Approach 
1.  HSDW  Data  Sources 


Transformational  Approach 
3.  Medical  Home  Data  Mart 


Medical  Home  Data  Mart: 

-  Business  Intelligence  Oriented 
Managed  Non-Normal  Form 

-  Retains  History 


€ 

IPW1® 


i  Encounter 
i  Appointment 
■  Diagnosis 
i  Procedure 
i  Population 
i  FTE  Personnel 
i  Facility  RVU 
i  Patient  Satisfaction 


m 
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Improve  Current  PCMH  Metrics 
ED/UC  Utilization 
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HSDW  Reporting  Options 


Create 
personalized 
reports,  charts, 
spreadsheets, 
and  dashboards 


Toolkit  Includes: 

Web  Report  Studio 
Bl  Dashboard 
Enterprise  Guide 
information  Maps/Cubes 


Integrity  -  Service  -  Excellence 
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In 


t  eg  r  it  y 


Service-  Excellence 
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Thank  You! 


Questions? 


Claudine.Ward@us.af.mil 

DSN  761-6115 
703-681-6115 
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The  Application  of  Johns  Hopkins  Adjusted  Clinical  Group  Case-mix  System  in  AFMS 

Ms  Susan  Chao,  Office  of  the  Chief  Information  Officer,  Air  Force  Medical  Support  Agency 

The  Adjusted  Clinical  Group  (ACG)  Case-Mix  System  is  a  diagnosis-  and  medication-based  risk-adjustment 
tool  that  has  been  adopted  by  more  than  200  healthcare  organizations  in  US  and  abroad  and  validated 
extensively  in  commercial  and  research  settings  over  15  years,  but  has  only  recently  been  implemented  in 
AFMS.  ACG  offers  a  comprehensive  family  of  measurements  designed  to  help  explain  and  predict  how 
healthcare  resources  are  delivered  and  consumed.  Through  its  unique  ‘person-focused’  approach,  ACG  captures 
the  multidimensional  nature  of  individual’s  health  and  morbidity  burden  of  patient  population,  and  it  also  can  be 
used  to  identify  and  predict  health  care  resource  needs,  enhance  equitable  distribution  of  limited  resources, 
improve  accuracy  in  provider  profiling,  streamline  healthcare  delivery,  evaluate  population  health  risk,  and 
provide  actionable  information.  FY09-FY10  M2  data  were  used  to  demonstrate  capabilities  of  ACG  and  to 
validate  its  predictive  models  in  AFMS-enrolled  population.  Sensitivity  of  predictive  models  for  high  total 
healthcare  cost,  high  pharmacy  cost  and  hospitalization  were  39%,  69%  and  28%,  respectively,  whereas  the 
corresponding  specificity  were  97%,  98%  and  96%,  respectively.  The  performance  of  ACG  in  AFMS  was 
comparable  to  that  found  in  commercial  HMO  populations  where  the  sensitivity  for  high  total  healthcare  costs 
and  hospitalization  were  37%  and  33%,  respectively.  This  suggests  that  ACG  can  be  applied  to  AFMS  even 
though  it  was  originally  developed  using  commercial  HMO  and  state  Medicaid  populations.  AFMS  leadership 
should  take  advantage  of  the  readily  available  measures  generated  by  ACG  and,  with  these  unparalleled  and 
comprehensive  measures,  in  turn  develop  effective  population  health  policies. 
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Headquarters  U.S.  Air  Force 

Integrity-  Service  -  Excellence 

The  Application  of  Johns  Hopkins 
Case-Mix  ACG  System  in  AFMS 

Susan  Chao,  MS,  GS-13 

Lt  Col  David  Carnahan,  MD  MSCE 

Clinical  Informatics  Division 
AFMSA/SG6H 

U.S.  AIR  FORCE 


U.S.  AIR  FORCE 


■  Background 

■  Introduction 

■  Validation 

■  Application 

■  Questions 


Outline 


Integrity  -  Service-  Excellence 


Background 

V 

❖ 

U.S.  AIR  FORCE 

U.S.  AIRFORCE 

■  Grew  out  of  Dr.  Barbara  Starfield’s  research  hypothesis: 

Clustering  of  morbidity  is  a  better  predictor  of  health  services 
resource  use  than  the  presence  of  specific  disease 

■  Conceptual  Basis: 

Assessing  the  appropriateness  of  care  needs  to  be  based  on 
patterns  of  morbidity  rather  than  on  specific  diagnoses 

Introduction 

■  Developed  by  the  Johns  Hopkins  School  of  Public  Health 

■  A  ‘person-focused’  comprehensive  family  of  measurement  tools 

■  Adopted  by  200+  healthcare  organizations  world-wide 

■  Case-mix  adjust  more  than  20  million  covered  lives 

■  Most  widely  used  &  tested  population-based  risk-adjustment  system 

Integrity  -  Service -  Excellence 

Integrity  -  Service-  Excellence 
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V 
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U  S.  AIR  FORCE 


Components 


Diagnosis-based  markers 


Pharmacy-based  markers 


Hospital  dominant  conditions 


Frailty  markers 


Predictive  modeling 


Care  coordination  markers 


Pharmacy  adherence  markers 


Input 


Data  Analysis 


Output 
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Diagnosis-based  markers: 

Two  views 


Morbidity  View 

-l 

Disease  View 

-I 

ADG 

(Aggregated  Otegnoal*  Group*) 


EDC 

(Expended  Diagnosis  Clusters) 


Classify  Dx  into  clinically  meaningful,  but  not  disease -specific,  morbidity  groups 
Classify  Dx  into  disease  specific  markers 


ICD-9 

ADG 

EDC 

4019 

Hypertension  nos 

10 

Chronic  Medical:  Stable 

CAR14 

Hypertension,  w lo  major 
complications 

4011 

Benign  hypertension 

10 

Chronic  Medical:  Stable 

CAR14 

Hypertension,  wto  major 
complications 

2724 

Hyperlipidemia 

necAios 

10 

Chronic  Medical:  Stable 

CAR11 

Disorders  of  lipid 
metabolism 
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Diagnosis-based  markers: 

Morbidity  view 


•  Duration 

•  Severity 

•  Diagnostic  Certainty 

•  Etiology 

•  Specialty  Care 


•Age 

•Sex 

•Specific  ADG 
•#  of  major  ADG 
•#  of  ADG 


Similar 

•  Needs  for  healthcare 
resources 
■  C  finical 
characteristics 


Examples: 


ADG 

ACG 

Time  limited:  major 

Likely  to  recur,  discrete 

Acute  minor  /  likely  to  recur,  age  6+,  w/o  allergy 

4-5  other  ADG  comb,  age  45+,  2+  major  ADGs 
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Diagnosis-based  markers: 
ACG  -  Concurrent  Weight  -  RUB 


ACG 

Description 


Assessment  of 
relative  resource  use 


Mean  cost  of  all  pt 
in  an  ACG  divided 
by  mean  cost  of  all 
pt  in  the  population 


0  =  None  User 

1  =  Healthy  User 

2  =  Low 

3  =  Moderate 

4  =  High 

5  =  Very  High 


RUB 

(Resource  Utilization  Band) 


Integrity  -  Service-  Excellence 


54 


Proceedings  of  the  2011  AFMS  Medical  Research  Symposium 

Volume  4  Healthcare  Informatics 


V 

❖ 

US.  AIRFORCE 


Diagnosis-based  markers: 
ACG  -  Concurrent  weight  -  RUB 


ACG 

Reference  Concurrent 
Weight 

RUB 

Commercial 

(0-64) 

Medicare 

(>=65) 

Acute  Minor,  Age  6+ 

0.16 

0.10 

1 

Chronic  medical:  stable 

0.35 

0.15 

2 

2-3  Other  ADG  combinations,  age  1-17 

0.50 

0.15 

2 

Acute  major/Likely  to  recur 

0.53 

0.24 

3 

Pregnancy,  2-3  ADGs,  1+ major  ADG 

2.64 

0.79 

4 

10+ Other  ADG  combinations,  age 

18+,  0-1  major  ADG 

3.32 

1.06 

4 

6-9  Other  ADG  combinations,  age  35+, 

3  major  ADGs 

6.89 

1.87 

5 
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Diagnosis-based  markers: 

Disease  view 


Similar  clinical 
Characteristics 


Provider 

specialty 


1 EDC  1 ICD-9 

IMEDC 

|  MEDC-type 

Diabetes  comp  DM  w  Ketoacidosis 

DM  w  coma 

DM  w  renal  manifestation 

Allergy 

Cardiovascular 

General  Surgery 

Administrative 

Medical 

Surgical 
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Special  population  markers 


■  A  set  of  Dx  codes  that 
indicates  a  pt  has  high 
probability  of  being 
hospitalized  in  the  next 
12  months 

A  set  of  Dx  codes  that 
describe  clinically  frail  pt 


J* 


Examples: 

Acute  renal  failure 
Acute  resp  failure 

Impaired  vision 
Dementia 


An  aggregate  marker  of 
case  complexity 


Obesity 
Hypertension 


I*  For  high  prevalence  p 

I  chronic  conditions  — ' 
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Predictive  modeling 
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Predictive  models 


Coordination  of  care  markers 


U.S.  AIR  FORCE 


•  Total  costs  ■  Pharmacy  costs  Probability  of: 


Prob.  high  total 
costs 

Predicted 
resource  Index 
(PRI) 


Prob.  high  pharmacy 
costs 

Pharmacy  PRI 

High  risk  for  unexpected 
pharmacy  costs 


IP  Hosp  12  mos 
IP  Hosp  6  mos 
ICU  Hosp 
Injury  Hosp 


•  Prob.  unexpected  Rx 
costs 


Extended  Hosp 


0  0  “ST 


Uvteltt  1  visit  1  visit 


Example: 

Endocrinologist 

4 

Cardiologist 

3 

Pulmonologist 

2 

Neurologist 

1 

Marker 

Value 

Majority  source  of  care 

40% 

Generalist  seen 

No 

Unique  provider  count 

4 

Unique  specialty  count 

4 
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Validation 
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Validity: 
Predictive  Models 

■ 

Sensitivity  (%) 

Specificity  (%)  | 

E 

AFMS 

|  HMO/ Medicare 

AFMS  f 

High  total  cost 

39 

37/24 

97 

High  Rx  cost 

69 

- 

98 

Hospitalization 

28 

33 

96 

Average  Total  Cost  by  RUB 

Average  Total  Cost  by  Concurrent  Weight 

Z 

*  - 

■ 

wm  r 

*  IO0CC 

=  0.83 

— !  -  - 

rail 

♦ 

-  "sr 

RUB 

Concurrent  Weight 
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Validity: 

Hospital  Dominant  Conditions 


Effects  of  Hospital  Dominant  Conditions  on  Next  Year’s  Outcomes 


■■ .  “  t  Next  Year’s  Outcomes 

Baseline  Year  Risk  Factor 

%  with  1+ 
hospitalization 

Mean 

total  healthcare  cost 

Mean 

pharmacy  cost 

Hospital  Dominant  Conditions 

AFMS 

US  pop 

AFMS 

US  pop 

AFMS 

US  pop  | 

<65  years  old 

None 

3.8 

4.2 

S3, 492 

$1,875 

$427 

$445 

1 

19.0 

20.4 

S16.272 

$12,652 

$3,181 

$2,342 

2+ 

39.9 

45.8 

$45,641 

$35,802 

$5,731 

$4,098 

>=65  years  old 

None 

15.0 

14.4 

$14,386 

$5,189 

$1,838 

$1,007 

1 

32.2 

35.3 

$34,997 

$14,810 

$4,203 

$2,112 

2+ 

47.9 

35.0 

$55,407 

$28,407 

$4,757 

$2,361 
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Application 


Integrity  -  S e rv i c e  -  E x c ell e n c e 


What  can  ACG  do  for  AFMS? 


U.S.AIR  FORCE 
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Population  profiling: 
Prevalence 

S.  AIR  FORCE 

Prevalence  of  common  chronic  conditions  by  age 


ur  I  I  118  44 

•  4S_« 

•IJJllJijNii 

y  s///*/// 

*  / 
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Population  profiling: 

RUB 


u.S.xiw  force 


Population  profiling: 
Movers  analysis 


Resource  Utilization  Band  by  MTF 


Noneuur  HaeMhvueer  Low  Moderate  HigH  VeryMglt 

Resource  Utilization  Band  (RUB) 


Tracking  morbidity  burden  over  time 


FY10  (case  mix=1.05) 

Lor  M«d»ir«1)t  Hijh 

Morbidity  Mosbldlry  Morbidity 

FY09 

liOW 

253,881  107,961  10,428 

(case-mix=1  01 ) 

Morbidity 

68.2%  29X3%  2.8% 

P: S422  P: S700  P:$783 

C:  S411  C$1893  C$6048 

Moderate 

100,832  210,843  51,664 

Mortbidiry 

| 

27.8%  58.0%  14.2% 

P:  SI ,783  P:S2,698  P:  $3,824 

C:$725  C$2,808  C:  $7,830 
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Provider  profiling: 
Disease  Burden 

U.S.  AIR  FORCE 


PCM  Specialty 

Family  practice  physician 

27 

35 

1.07 

Physician  assistant 

15 

22 

0.98 

Pediatrician 

ii 

15 

0.44 

Nurse  practitioner 

8 

7 

1.02 

Internist 

7 

4 

3.01 

RUB  by  PCM  Specialty 

50 
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MTF  profiling: 
Case-mix  adjust  performance 

U.S.  >IR  FORCE _  _ _ 


MTF 

FY09 

Case-mix 

FY10  Admission  Rate  (%) 

Unadjusted  Adjusted 

Little  Rock 

0.76 

3.4 

4.0 

Elmendorf 

0.85 

3.8 

4.3 

Robins 

1.11 

4.3 

4.1 

MacDill 

1.29 

6.2 

5.6 

Wright-Patterson 

1.49 

6.6 

5.3 
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Disease  management 


Percent  of  Patients  Identified  by  ICD  or  NDC  or  Both 


Congestive  heart  failure 


^  ^ _ 

(Average  Predicted  Probabilities  by  RUB 
for  patients  with  diabetes 

1.  l 


Age  and  Sex  Adjusted 
Standardized  Morbidity  Ratio 
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Case  management: 
Care  coordination 


Identify  Patients  at  Risk  for  Poor  Care  Coordination 

Number  of  Patients 

at  High  Risk  for  Poor  Care  Coordination*  Sample  Patient  Profile 


111 


liilill 

'  */  ■?/  s/Z 


Sex 

F 

Age 

70 

#  Chronic  condition 

8 

#  unique  providers  seen 

10 

^unique  specialists  seen 

7 

Generalist  seen 

N 

Majority  source  of  care  (%) 

24 
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Case  management: 
Frail  patients 


U.S.  AIW  FORCE 


Resource  allocation 


Identify  Frail  Patients  with  Risk  of  Injury-related  Hospitalization 


Number  of  Frail  Patients 

with  more  than  20%  risk  of  injury-related  hospitalization  Sample  Patient  Profile 


Z  S//  *//  s  /  * 


Sex 

F 

Age 

87 

#  Chronic  Condition 

9 

#  Hosp  Dorn  Condition 

3 

#  ER  visits 

2 

#  IP  admissions 

2 

#  OP  visits 

65 

Comparison  of  Characteristics  Affecting  Physician  Productivity 


Dr.  A 

Dr.  B  1 

Dr.  C 

Dr.  D 

Dr.  E  1 

#  of  Enrolled  pt 

2498 

1374 

1535 

1150 

1354 

Average  pt  age 

33 

38 

44 

49 

39 

%  Female 

45 

44 

52 

46 

43 

Case-mix 

0.86 

1.10 

1.10 

1.59 

1.11 

%  pts  w  >=1  hosp  dom  condition 

1 

1 

2 

3 

2 

%  pts  w  >=3  chronic  conditions 

7 

10 

15 

24 

10 

%  pts  w  frailty  condition 

2 

3 

4 

5 

3 

%  pts  w  >2  major  ADGs 

2 

3 

3 

7 

3 

Average  #  of  EDC 

6.0 

6.6 

6.5 

8.0 

6.7 

Average  #  of  Rx-MGs 

4.3 

4.9 

4.6 

5.9 

4.8 
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Questions? 

Contact:  susan.chao@us.af.mil 

Comprehensive  slide  set  available  upon  request 
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Pediatric  Critical  Care  Training  Validation  Using  High-Fidelity  Pediatric  Simulation 
Lt  Col  (Dr)  Daniel  Bruzzini,  711  HPW/USAFSAM-ETS 

Purpose:  First-year  pediatric  residents  and  those  deployed  to  natural  disasters,  humanitarian  crises,  and 
counterinsurgency  battlefields  must  have  the  capability  of  treating  children  with  critical  care  needs. 

Hypothesis:  Teaching  the  Society  for  Critical  Care  Medicine’s  (SCCM’s)  Pediatric  Fundamentals  of  Critical 
Care  Support  (PFCCS)  course  and  incorporating  high-fidelity  simulation  pediatric  critical  care  scenarios  will 
improve  the  fund  of  knowledge,  self-confidence,  and  performance  capability  of  first-year  pediatric  residents. 

Methods:  All  pediatric  residents  at  the  St.  Louis  University  School  of  Medicine  and  the  University  of  Missouri 
were  taught  the  SCCM  PFCCS  Course.  Each  student  completed  an  SCCM  standardized  and  validated  pretest 
and  posttest,  a  survey  of  10  five-point  Likert  scale  questions  on  managing  critical  children  before  and  after,  and 
2  videotaped  pediatric  critical  care  simulations  with  debriefings  after  each  scenario. 

Results:  Fund  of  knowledge  improved  from  a  pretest  score  of  60%  to  a  posttest  score  of  80%.  Pediatric 
residents  reported  feelings  of  preparation  increased  by  an  average  of  0.97  points  on  the  Likert  scale.  Ten  of  1 1 
pediatric  residents  indicated  they  thought  the  course  was  “extremely  helpful.”  Pediatric  critical  care  simulation 
time  to  recognize  a  failed  airway  went  from  72  s  to  46  s.  The  time  to  perform  CPR,  defibrillate  with  paddles, 
and  give  intravenous  epinephrine  decreased  from  3.50  to  1.33  min. 

Conclusions:  Pediatric  critical  care  fund  of  knowledge,  self-confidence,  and  clinical  performance  were 
improved  in  pediatric  first-year  residents  by  the  SCCM  PFCCS  Course  with  high-fidelity  simulation,  thereby 
validating  it  as  an  important  training  methodology  in  building  pediatric  critical  care  capability. 
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IM.  AIM  POM 


Pediatric  Critical  Care  Training  Validation 
Using  High-Fidelity  Simulation 


Col  (s)  Daniel  B.  Bruzzini,  MD 
C-STARS  Director  of  Pediatric  Intensive 
and  Emergency  Medicine 

Every  Airman  a  F orce  Multiplier 
August  2011  AFMS  Research  Symposium 


Dlsroutlon  Statement*  ahto-m  fiy  puDH:  release:  atsntuton  isirurmea  Case  Numoer  ESABW-23ii-ase4.25  jul  2D11 


Disclosure  ;&» 

Every  Airman  a  Force  Multiplier 


v  l  have  no  relevant  financial  interests  to  disclose. 


v  The  views  expressed  in  this  presentation  are  solely 
those  of  the  author  and  do  not  reflect  the  official 
policy  of  Saint  Louis  University,  SSM  Cardinal 
Glennon  Children’s  Medical  Center,  the  Department 
of  the  Air  Force,  Air  National  Guard,  Department  of 
Defense,  or  the  United  States  Government. 


□Etieutun  Statement  A.  Approved  far  pjaic  reease:  dtsticuton  is  trtriRed  Case  Numoer:  SSABW-201 1  -4064, 25  Jul2Gii  2 


Overview 


Every  Airmen  a  Force  Multiplier  ■ 


PFCCS  Notable 
Military  Applications 

Every  Airman  a  Force  Multiplier 


Pediatric  Fundamentals  of  Critical  Care  Support 
(PFCCS)  Course 

v  Interactive  Case  Study  Directed  Didactics 
v  Small  Group  Skill  Stations 
v  High-Fidelity  Simulations 

v  Validating  PFCCS  Educational  Research  (KCC  Method) 
v  Knowledge  Acquisition 

v  Confidence  in  Caring  for  Critically  Ill/Injured  Children 
v  Competency  Assessment  of  Knowledge/Skills  Taught 


v  i»t  pfccS  course  held  at  the  Air  National  Guard’s 
Readiness  Frontiers  National  Meeting  June  2011 
v  CERFP  -  CBRNE  Enhanced  Response  Force  Package 
v  HRF  -  Homeland  Response  Force 

v  Travis  AFB  rapid  response  team  requirement  for  its 
pediatricians 

v  Iraq/Afghanistan  training  requirement  for  deploying 
Air  Force  pediatricians 


Otstnoullon  Statement*  Approved  for  puoiis  retease:  dlstttuton  is  urarreea  Case  Numoer  88A3W-2Q1 1-4061. 25  jui  201 1  3 
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PFCCS  Mission 

h  Every  Airman  a  Force  Multiplier  M 


PFCCS  Content 

Every  Airman  a  Force  Multiplier  m 


v  Prepare  nonintensivists  for  pediatric  critical  care 
v  Physicians/surgeons 
v  Residents 
v  Physician  assistants 
v  Nurse  practitioners 
y  N  u  rse  s  Pediatric  Fundamental 

y  Resp irato ry  the ra pists  Critical  Care  Support 

y  Critical  care  EMTs/paramedics 
v  Transport/Air  Evacuation  Teams 


v  Core  subjects 

v  Initial  Approach  to  Pediatric  Trauma 
y  Airway  Evaluation  and  Management 
y  Cardiovascular  Shock  and  Management 
v  Mechanical  Ventilation  Management 
y  Sedation,  Analgesia,  and  Paralytic  Usage 
y  Transport  of  Critically  ill  Child 


v  Society  of  Critical  Care  Medicine  (SCCM) 


Elstnoullon  SStefrertA.  Approved  Tor  puoK  mease;  aetlsulor  Is  LtWRea.  Case  Nuirtrer  3SA3W-23il-«64,  IS  Jul  201 1 


ClstrojtBr  SBteTert  A  Approved  for  puMic  release;  denoulon  Is  inumed.  Case  Numoer  SSASW-Ijn-aoea.  25  johoii 


PFCCS  Content  (cont.) 


■  Every  .-iirman  a  Force  Multiplier  l 


V 

♦I* 


Case  Study  Directed 
Didactics 


■  Every  Airman  a  Force  Multiplier 


v  Core  subjects 

y  Arterial  Blood  Gas  Interpretation 
v  Post-Operative  Care 
v  Fluid  and  Electrolytes 
v  Acute  Renal  Failure 
v  Neurological  Emergencies 
y  Diagnosis  and  Management  of  Acute  Infection 


BtsOKutlon  Statement  A  Approved  TOrpuslte  release;  dfensuion  is  LrUTKea.  Case  Numoer:  3SA9W-2Cll-«64, 2S  JuHOil 
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Clinical  Skill  Stations 

Every  Airman  a  Force  Multiplier 


Mechanical  Ventilation 


■  Every  Airman  a  Force  Multiplier  I 


Small  groups  of  6-8  students 
v  Airway  management 
Mechanical  ventilation 

v  Train  on  ventilators  used  in  practice  or  in  deployed  setting 
Transport  emphasizing  care  and  proper  documentation 
Sedation,  analgesia,  and  paralytics 
v  Medication  usage 
v  Morphine  PCA  usage 

v  Invasive  device  pediatric  critical  care  X-ray/CT  review 


Ctstnoutlon  Statement  A  Approved  Tor  puolK  release;  atertsulon  is  inarmed.  Case  Nuiroer  3&ASW-2011-4064,  15  jul  201 1  8 


3l6Woutlon  Statement  A:  Approved  for  puone  reease;  otsrmuiort  is  inarmed.  Case  Muiroer  5SASW-2Gll-«64, 25  JUI2011  ID 


Mechanical  Ventilation 

Every  Airman  a  Force  Multiplier 


V 


PFCCS  High-Fidelity 
Simulation 


Every  Airman  a  Force  Multiplier 


asmouttor  statement  a  Approved  forpuoitc  reease;  aerttulor  is  intuited.  Case  Nutnoer  3SA9w-2Cii-«64,2S  juI2C11  11 


Two  high-fidelity  simulations 

v  Pediatric  intensivist  video  analysis  and 
debriefing 

v  Competency  areas 

v  Recognition  and  treatment  of  dysfunctional 
airway 

v  Recognition  and  treatment  of  fatal  arrhythmias 
v  Appropriate  use  of  CPR,  defibrillation,  and 
epinephrine 

Ctstiouttor  Statement  a  Approved  tor  puone  release;  dtsttomcri  is  inarmed.  Case  iNWmoer.  38A0W-2311-4O64, 25  jul  2di  13 
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PFCCS  Educational  Research 

Every  Airman  a  Farce  Multiplier 


PFCCS  Validation  Process 
KCC  Methodology 

Every  Airman  a  Farce  Multiplier 


v  Drs.  Bruzzini,  Werner,  Garrett,  and  H.  Sallee 

v  Institutional  Review  Board  approved 

Study  Groups 

v  1st  year  pediatric  residents  from  Saint  Louis  University 
and  University  of  Missouri  at  Columbia 
v  3rd  year  pediatric  residents  from  Saint  Louis  University 


Elstmxtlon  SSteirertA.  Approved  Tor  puonc  reease;  ostlsulor  Is  Lnnntea  Case  Nutroer  55A3W-23il-«64,  2S  Jui  201 1  14 


v  Knowledge  Acquisition 
v  Pre-  and  post-PFCCS  tests 
v  Confidence  in  Caring  for  Critically  Ill/Injured 
Children 

v  Pre-  and  post-course  surveys 
v  Clinical  Competency  and  Integration  Assessment 
v  Group  improvement  from  simulation  1  to  number  2 
v  Compared  performance  to  traditionally  trained  3rd  year 
residents  who  had  never  taken  PFCCS 


CIsTAitlon  Statement  A  Approved  Tor  puwic  release;  dtsrtnulon  Is  manned.  Case  Nutroer  53A3W-2D1 1-4064, 25  joI20ii  15 


Every  Airman  a  Force  Multiplier 


Must  preread  11  of  24  PFCCS  textbook  chapters 


v  Designed  to  be  used  for  future  reference  textbook 
v  PFCCS  pre-test  and  post-test 
v  Must  score  70%  or  better  to  earn  certification 


CIsmoulMn  statement  A.  Approved  TOrpuMlc  release:  dtsimulon  is  tmrrtteo.  Case  Numoer:  3SASW-2Dil-«64, 25  JUI2B11  16 


Test  Results 

Every  Airman  a  Farce  Multiplier 

V  Pre-test  N=42  v  Post-test  N=42 

V  Mean  57.9  Std.  14.2  v  Mean  83.9  Std.  7.4 

Pre-test  -  Post-test  difference  Mean  26  Std.  12.6 
95%  Cl  22.1  -29.9,  p-value  <  0.0001 


■-Test 


Ctstrouttin  Statement  A  Approved  tor  puoie  release;  dlstisuion  is  manned.  Case  Numoer.  38Aew-20ii-4064, 25  Jul  2011  17 
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Pediatric  Critical  Care 
Confidence 

Every  Airman  a  Force  Multiplier 


Pediatric  Critical  Care 
Confidence 

Every  Airman  a  Force  Multiplier 


v  Pre-/post-surveys 
v  10-question,  5-point 
Likert  scale 
v  1  poorly  prepared  to 
5  extremely  prepared 

v  Ascertain  feelings  of 
competency 
v  Overall 
v  Nine  key  areas 


v  Overall  pediatric  crit.  care 
'  Monitor  intubation 
Operate  a  ventilator 
v  Choose  ventilator  modes 
vRSI  medication  usage 
Lead  a  medical  code 
v  Correct  electrolytes 
-Arrange  peds.  transport 
v  Provide  sedation/analgesia 
Manage  pediatric  shock 


Elstnoullon  Statemert  A  Approved  for  puoltt  reease;  aistloulon  Is  irttmea  Case  Murroer  8&A9W-2011-4064,  IS  Ail  2011  *8 


vSignificant 

v  Overall  p  <  0.001 

v  Choose  vent  mode  p  <  0.002 
vRSImed.  use  p<  0.016 
v  Lead  medical  code  p  <  0.017 
v  Correct  electrolytes  p  <  0.0001 
v  Sedation/analgesia  p<  0.012 
v  Manage  ped.  shock  p  <  0.001 


vNot  Significant 

v  Monitor  intubation 

v  p  <  0.06 

v  Operate  a  ventilator 

V  p  <  0.1 

-Arrange  peds.  transport 

v  p  <  0.12 


*  Pared  samples  i-Test 

Ilstrdjtnr  State  rent  A  Approved  for  puolte  teease;  aertoulor  is  inlimed.  Case  Nutrtrer  5SASW-2011-4064. 25  JUI2D11  20 


Clinical  Competency 


■  Every  Airman  a  Force  Multiplier  l 


Simulation  Scenarios 

Every  Airman  a  Force  Multiplier 


v  Two  physiologically  identical  critical  care  scenarios 
v  #1  -  Respiratory  failure  due  to  pneumonia 
v  Displaced  endotracheal  tube 
v  Ventricular  fibrillation 
v  #2  -  Post  operatively  intubated  patient 
-  Obstructed  endotracheal  tube 
-Pulseless  ventricular  tachycardia 

Normal  Sinus  Rhythm 


'  *  1 


Ventricular  Fibrillation  Ventricular  Tachycardia 

QteDButun  SBtemeittA:  Approved  for  puone  release;  dfcTlsuion  Is  ir*med.  Case  Nutnoer  58ASW-23ll-«64, 25  jul  2011 
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Simulation  Scenarios  5T 

❖ 

Every  Airman  a  Force  Multiplier 

v  Standard  scenario  read  to  all  in  the  beginning 
v  All  vital  signs,  transitions,  high-fidelity  simulation 
responses  to  student  inputs  programmed 
v  Simulation  operators  behind  1-way  glass 


v  No  verbal  or  nonverbal  feedback  during  simulation 
v  No  adjustment  of  simulation  physiology  by  sim.  operator 


Distnouflon  ststemert  a  Approved  for  pusns  teease,  aeriauior  is  inarmed  case  Nurroet  3SASW-2art-M6«, 25  jul20ii  23 


Simulation  Scenarios 

❖ 

E\ery  Airman  a  Force  Multiplier 

-  All  necessary  equipment  laid  out  on  a  table  for  use 
v  Avoid  time  delays  due  to  inability  to  find  equipment 
v  Team  leader  assigned  and  wore  a  badge 


v  Avoids  no  leader  or  too  many  leaders  situations 

-  Same  group  but  different  team  leader  for  second  scenario 


Simulation  Results 

Every  Airman  a  Force  Multiplier 


v  Compared  Simulation  1  to  Simulation  2  performance 

v  Delineates  simulation  and  debriefing  impact 


v  Compared  3rd-year  residents  to  1st-year  residents 
v  3rd  year  residents  have  3  mo  of  Pediatric  ICU  experience 
v  Determines  PFCCS  course  effect 


ClstrautBn  sate  Tent  A  Approved  for  puolko  re  ease;  d&rioulcn  is  imritea.  Case  Numoer  5SAav-23l1-«64. 25  jul  2011  26 
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Simulation  Results  Measured  % 

❖ 

Every  Airman  a  Force  Multiplier 

v  8  groups  of  1st-year  pediatric  residents 
v  New  method 

v  3  groups  of  3rd-year  pediatric  residents 
v  Control 
v  Total  time 

v  Time  to  disconnect  from  ventilator 
v  Time  to  pull  endotracheal  tube  (ETT) 
v  Time  to  recognize  the  arrhythmia 
v  Time  to  begin  cardiopulmonary  resuscitation  (CPR) 
v  Time  to  deliver  first  electrical  shock 
v  Time  to  deliver  first  dose  of  epinephrine 

Elstflouflor  sateirent  A.  Approved  for  puollc  reeaBe;  dBtlaulon  Is  ixArnea  Case  Mu  freer  5aA3W-23il-«6i,  IS  Ail  201 1  27 


Simulation  1  vs.  2 

*;*  1st-Year  Pediatric  Residents 

r 

SIMULATION  1 

SIMULATION  2 

v  Total  time  p  <  0.001 

v  Total  time 

v  10  min  22  s  (622  s) 

v  6  min  59  s  (419  s) 

v  Disconnect  vent  p<0.042 

v  Disconnect  vent 

v  70.5  s 

V  52.25  s 

v  Pull  ETT  p<  0.004 

vPull  ETT 

v3  min  26  s  (206  s) 

v  1  min  40  s  (99.6  s) 

Clstrojter  saiarerl  a.  Approved  for  puMic  refe36e;  ostttulon 

IS  uwnted.  Case  Muiroer  3SA0W-2Oii-«64. 25  juHGil 

28 

Simulation  1  vs.  2  P® 

V  1st-Year  Pediatric  Residents 

SIMULATION  1 

SIMULATION  2 

v  ID  arrhythmia  p  <  0.9 

vID  arrhythmia 

v  16.9  s 

v  7.5  s 

vStartCPR  p  <  0.8 

v Start  CPR 

v*  40.1  s 

V  35.9  s 

v  1st  shock  p<0.47 

v  1st  electrical  shock 

v  91.5  s 

v  56.4  s 

v1stepi.dose  p<0.11 

v  1st  epinephrine  dose 

^  2  min  34  s  (154  s) 

v  1  min  29  s  (89  s) 

Clsimutar  SaterentA  Approved  for  puolfc  teiease;  dlswsuion  Is  iramed.  Case  Mjmoer  38A0W-2an-«64, 25  jul  2011  80 
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Simulation  1  vs.  2  1st-  and 
V  3rd-Year  Pediatric  Residents 

Every  Airman  a  Force  Multiplier 


v  Same  improvement  rate 
in  total  simulation  time 
v  Defines  simulation  effect 

v  Bench  mark  analysis 
v  Best  3rd-year  score  vs.  1st 
year’s  mean  score 
v  510  s  versus  419  s 
p  <  0.003 

-  Mann-Whitney  non- 
parametric  test  p  =  0.06 


Mann-WYntneyn<vi-C3rame&c  tea 
Avoid  Mest  oulfcer  effect  on  nean 


Dletrautlan  Statement  A  Approved  tor  puolic  refe3se;  dBimulon  Ie  uwrlted.  Case  Muiroer  5SABW-2011-4064. 25  JUI2D11 


1st-Year’s  Sim  Results  vs. 
3rd-Year’s  2nd  Sim  Results 


Exwry  Airman  a  Force  Multiplier 


1st-Year’s  Sim  2 

V  NEW  METHOD 
v  Total  time  p  =  0.06 
v  6  min  59  s  (419  s) 
v  Disconnect  vent 
v  52.25  s  p<0.01 
-  Pull  ETT  p  <  0.22 
v  1  min  40  s  (99.6  s) 


3rd-Year’s  Sim  2 

v CONTROL 
v  Total  time 

v  9  min  2  s  (542  s) 
v  Disconnect  vent 
v  1  min  17  s  (76  s) 
vPull  ETT 

v  2  min  43  s  (163  s) 


SlEtneutlon  SStemertA.  Approved  for  puoic  isease;  OCTBsuian  i»  inarmed.  Case  Nutroer:  5SA3W-2Cii-«64, 25  JUI2011 
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1st-Year’s  Sim  Results  vs. 
3rd-Year’s  2nd  Sim  Results 

Every  Airman  a  Force  Multiplier 


1st-Year’s  Sim  Results  vs. 
3rd-Year’s  2nd  Sim  Results 

Every  Airman  a  Force  Multiplier 


Time  to  Give  1st  Dose  of  Epinephrine 


Control:  3rd  Year’s 

SOD  - 


1  ^-Year’s  Sim  2 
v  Identify  arrhythmia 
v  7.5  s  p  <  0.9 
/Start  CPR 
v  35.9  s  p  <  0.84 
v  1st  electrical  shock 
v  56.4  s  p  <  0.26 
v  1st  epinephrine  dose 
v  1  min  29  s  (89  s)  p  <  0.1 


3rd-Year?s  Sim  2 
v  Identify  arrhythmia 
v  12  s 

V  Start  CPR 
v  48.3  s 

y  1st  electrical  shock 
v  1  min  27  s  (87  s) 
v  1st  epinephrine  dose 
v  3  min  (180  s) 


Mann-Whrtney  r.orv-paramemc  test 


New  Method:  19t  Year’s 


i  inuntea.  Case  Nutroer  58A0W-2Dn-«64, 25  juI2B11  40 


Cistnajtfcr  statement  a.  Approved  for  puonc  reease;  aetiauior  is  unarmed  case  Number  ssasw-20ii-«64,  2S  jul  201 1  30 


Conclusions 

■  Every  Airman  a  Force  Multiplier  1 


S' 


v  Knowledge  acquisition  improved 

v  Post-test  scores  mean  increase  26%  (95%  Cl  22%-30%) 

v  Confidence  significantly  improved  (all  p  <  0.02)  in: 
v  Caring  for  Critically  III  Children 
v  Choosing  the  Appropriate  Ventilation  Mode 
v  Rapid  Sequence  Intubation  Medication  Usage 
v  Leading  a  Medical  Code 
v  Correcting  Electrolytes 
v  Providing  Sedation  and  Analgesia 
v  Managing  Pediatric  Shock 


DlsmeuKin  statement  A.  Approved  for  puai:  reease;  dWtoulon  is  Lmrmea  case  Ni 


er:  3SA3W-2311-M64. 25  JUI2B11 


Conclusions  Simulation  1  to 


*:* 


Simulation  2 

■  Every  Airman  a  Force  Multiplier  l 


5* 


v  Simulation  effect  most  evident  in  airway  management 
v  Disconnecting  from  the  ventilator 
v  Pulling  the  endotracheal  tube 

v  Not  significantly  improved  in  dysrhythmia  therapy 
v  Initiating  CPR 
v  Defibrillation 
v  Epinephrine  administration 

v  Taught  in  Pediatric  Advanced  Life  Support  (PALS) 

vAII  pediatric  residents  PALS  certified 


ClstroutBr-  Statement  A:  Approved  for  puolte  release;  dferiouion  Is  Lrtrmed.  Case  Njtnoer  38ASW-23ii-«64, 25  JUI2G11 
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1st-Year  New  Method  vs. 
3rd-Year  Control 

Every  Airman  a  Force  Multiplier 


Conclusions 
PFCCS  Validation 


■  Every  Airman  a  Force  Multiplier  ■ 


v  Significantly  decreased 

v  Total  time 

v6  min  59  s  vs.  9  min  2  s 
v  Time  to  disconnect  from  vent 
v52  S  VS.  76  S 

v  Time  to  1st  epinephrine  dose 
vl  min  29  s  vs.  3  min 

v  Experience  -  negative  influence  on  airway  management? 
v  Give  fluid  boluses 
v  Request  x-ray  evaluation 
v  Limited  3rd -year  group  sample  size 

Dlswoutton  Statement  A.  Approved  lor  pusnc  teease;  dBtttwlor  Is  Lrttrneo.  case  Mutrosr  55A5W-23il-«64, 2S  Jul  201 1  43 


v  KCC  Course  Validation  Methodology 
v  Knowledge  Testing 
v  Pre-  and  post-tests 
v  Confidence  in  Capabilities 

vPre-  and  post-course  surveys 

v  Competency  Demonstration 

v  Properly  designed  high-fidelity  simulation 

v  PFCCS  is  a  valid  training  methodology  to  prepare 
for  children  with  critical  care  needs. 

Cl6tnoutian  statererl  a.  Approved  for  puolic  isBase;  dBrculon  Is  uvrrfted  Case  Nutnoer  38A0W-2O1 1-4064.  25  Ml  2011 


Thank  you! 

Every  Airman  a  Force  Multiplier 

v  Without  the  help  and  support  of  many  people,  my  SCCM 
PFCCS  course  would  not  have  happened: 


Hospital  President  Sherilyn  Hailstone,  SSM  Cardinal 
Glennon  Medical  Chief  of  Staff  Dr.  Wilmott,  Univ.  of  Mo 
Medical  Chief  of  Staff  Dr.  Fete,  Dir.  of  Nursing  Jeannie 
Mollohan,  ER  and  ICU  Department  Heads  -  Dr.  Flood,  Dr. 
Ream,  Instructors  -  Dr.  Garrett,  Dr.  Werner,  Dr.  Jamshidi,  Dr. 
Doerhoff,  Dr.  Scalzo,  Dr.  H.  Sallee,  Dr.  Lynch,  Anne 
Gildehaus  CCRN,  Jessica  Leet  CCRN,  Respiratory 
Therapists  Mary  Fairchild,  Brian  Eggenmeyer, 

Administrative  -  Erin  Donovan,  Rita  Stites,  Deane  Sagehorn, 
Barb  Beguette,  Nanette  Parris,  Kristy  Deutchmann,  Tammi 
Mooshigian,  Simulation  -  2nd  Lt.  Xiaoming  Shi  (AF  HPSP 
Med.  Student),  Dave  St.  Andre,  Mike  Kaufmann, 

Ellen  Pringle 

v  Statistical  Support  -  Eric  Armbrecht,  Ph.D. 


Clsmojtttn  sstemertA.  Approved  forpuai:  leease;  dCTtsuian  le  uurmea  Case  Ni 


er:  36A3W-2:i  1-4364,  25  JUI2C11 


71 


Proceedings  of  the  2011  AFMS  Medical  Research  Symposium 

Volume  4  Healthcare  Informatics 


Future  C-STARS  St.  Louis 
PFCCS  Courses 

Every  Airmen  a  Force  Multiplier 


Fridays  and  Saturdays 
v  23  &  24  September  2011 
v  14  &  15  October  2011 
v  27  &  28  April  2012 
v  11  &  12  May  2012 

^  Free  to  those  attending  C-STARS  St.  Louis 

v  Must  coordinate  in  advance  with  Col  (s)  Bruzzini,  MD 
*•  Reduced  rate  if  only  taking  the  PFCCS  course 
v  Contact:  pfccs.saintlouis@gmail.com  to  register 
v  Can  be  exported  off -site  to  train  personnel  locally 
v  Contact:  Col  (s)  Dan  Bruzzini,  MD  -  dbruzzin@slu.edu 


Dlswxttor  SBtemertA.  Approred  fwpuooc  reease;  aetHwloi  Is  unarm*!  case  Number  HASW-2011-4064. 15  Ail  2011  45 


Military/Civilian  Costs 

Every  Airman  a  Force  Multiplier 


v  No  charge  if  enrolled  in  C-STARS  St.  Louis  concurrently 
Must  coordinate  with  Col  (s)  Bruzzini  in  advance 
v  If  just  wish  to  take  the  PFCCS  without  C-STARS  training: 
V  Civilian  Physician/P.A./Nurse  $400/300/300 

v  Military  Physician/P.A./Nurse  $350/250/250 

v  All  Resident/R.T./Paramedic  $250 

v  Water  Tower  Inn  Hotel:  $75/night 

v  Walking  distance  from  course 
v  Includes: 

SCCM  PFCCS  Course  license 
v  SCCM  Consultant  Director,  and  Instructor  honorariums 
v  $80  PFCCS  textbook 


v  19-20  continuing  education  hours  for  all  students 
v  Rental  of  ventilators  and  their  compressors 
v  High-Fidelity  Simulation  expendable  supplies 
v  Breakfast/lunch/hospital  parking 

v  Contact:  pfccs.saintlouis@gmail.com  to  register  or  to  request  more 
information 


Diarajllon  SBtementA.  Approved  tor  puWK  release;  dBMDulon  is  urslmlted.  Case  Number  58Aew-23il-«64, 25  JUI2011  46 


Notables  on  St.  Louis  PFCCS 


Every  Airman  a  Force  Multiplier 


v  5th  institution  in  the  United  States  to  offer  the  Society 
of  Critical  Care  Medicine's  PFCCS  course 
v  1st  institution  to  incorporate  it  en  bloc  into  its  Pediatric 
and  Internal  Medicine/Pediatric  Residency 
v  1st  institution  to  validate  the  effectiveness  of  PFCCS 
residency  incorporation  through  educational  research 
*  The  only  PFCCS  institution  to  incorporate  high-fidelity 
simulations  with  full  videotaped  debriefings 
v  Highest  PFCCS  continuing  education  credit  nationwide 
v  Only  PFCCS  to  offer  continuing  education  hours  for 
respiratory  therapists  and  paramedics 


Cianoutlan  StatementA.  Approved  Mr  puMle  release:  dttlBUKr  le  irttmea.  Case  Number:  5SASW-2Cll-«64  25  jul  2011  50 
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Contemplating  a  New  Model  for  Air  Force  Aerospace  Medical  Technician  Skills  Sustainment  Training 

SMSgt  Robert  Corrigan,  59th  Medical  Wing,  United  States  Air  Force  Medical  Service 

Two  decades  ago,  Aerospace  Medical  Technicians  received  robust  skills  sustainment  training  through  exposure 
to  multi-faceted  patient  treatment  environments.  Available  training  environments  included  inpatient  care, 
outpatient  care,  and  emergency  services.  This  diverse  training  environment  made  possible  through  large 
operating  budgets  and  an  extraordinary  infrastructure  could  not  last.  Today  (after  five  separate  base  closure 
and  realignment  initiatives),  medical  funding  and  infrastructure  is  but  a  shadow  of  what  it  once  was.  Budgetary 
constraints  and  rising  healthcare  costs  have  necessitated  a  purposeful  movement  away  from  inpatient  and 
emergency  care,  toward  outpatient  and  preventative  medicine.  While  changes  in  Air  Force  health  care  delivery 
are  necessary,  the  closure  of  inpatient  and  emergency  services  throughout  the  Air  Force  Medical  Service 
significantly  impacts  our  ability  to  prepare  medical  professionals  and  paraprofessionals  for  deployed  operations. 
This  research  uses  a  mixed-methods  framework  (qualitative  and  quantitative)  to  demonstrate  the  importance  of 
exploring  alternative  training  models  for  medical  skills  sustainment  training.  Further,  the  study  suggests  an 
alternative  training  model  that  leverages  existing  network  technologies  (high  fidelity  patient  simulation, 
asynchronous  learning  networks,  and  video-teleconferencing)  to  satisfy  established  learning  objectives  in  the 
cognitive,  affective,  and  psychomotor  domains  of  learning.  The  proposed  model  offers  a  potential  mitigation 
strategy  for  medical  skills  sustainment  training  limitations  experienced  in  a  post-BRAC  era  plagued  by 
budgetary  constraints  and  the  near  complete  loss  of  inpatient  and  emergency  services  training  platforms. 
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Reason  for  Topic 


4«r 


Contemplating  a  New  Model  for  Air 
Force  Aerospace  Medical 
Technician  Skills  Sustainment 
Training 


IRFOw  L  SMSgt  Bob  Corrigan 


Today,  the  majority  of  Air  Force  Aerospace 
Medical  Technicians  perform  their  duties  in 
a  clinical  environment.  The  difficulty 
experienced  in  providing  effective  war-time 
readiness  training  for  these  technicians  has 
become  a  primary  concern  for  Air  Force 
senior  medical  planners. 


Strength  Through  Education 


Problem  Statement 


Research  Framework  (Background) 


s 


Inspectors  assert  we  should  not  experience  any 
problems  conducting  training  in  the  current  Air  Force 
medical  training  environment  (“because  nothing  has 
changed  in  our  training  methods  over  the  last  20  years”). 

I  suggest  that  while  training  methods  have  not  changed, 
everything  encompassing  the  Air  Force  medical  training 
environment  has  changed.  This  being  the  case,  I 
propose  new  methodologies  should  be  leveraged  to 
maximize  knowledge  transfer  in  our  present  “resource 
constrained”  training  environment. 


•  What  has  changed  with  regards  to  the  Aerospace  Medicine 
Technician  Training  Environment? 

a.  Personnel  Drawdowns  (Military  Implosion) 

b.  Base  Closures  (loss  of  facilities) 

c.  Movement  toward  outpatient  and  preventative 
medicine 
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W 


Sequential,  Exploratory  Design 
(Cresswell,  2003) 


m 


Qualitative 


A 

V 


quantitative 


Qualitative  Da:a 

- N 

Qualitative  Da:a 

- IS 

Quanttative  Da:a 

- N 

Collecticn 

- 

Analysis 

- 1/ 

Collecticn 

- 

V 


Seq.  Exp.  Design  (X2) 


\  Iteration  j 


Qualitative  Data 

-A 

Qualitative 

— N 

Quantitative 

A 

Quantitative  Data 

Lessons  Learned 

-y 

Content  Analysis 

“1/ 

Survey  Data 

“1/ 

Analysis  (ANOVA) 

^  Embedded,  Single  Case  Revelatory  gra 
**  Case  Study 

Sy  Literature  Review 

•  Embedded,  Single  Case  with  additional  sub-units 

•  Revelatory  in  nature  -  investigated  something  new 

•  Qualitative  with  quantitative  support  (Yin,  2006) 

•  Mixed  Methods  (Cresswell,  2005) 

7 

•  Limited  availability  of  literature  dealing  with  4N0s 

•  Over  200  peer  reviewed  references 

•  75  referenced  in  the  work  /  made  bibliography 

•  Included  Congressional  testimony  and  reports 

8 
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Methodology 


Completed  using  the  five  necessary  components  of  Case  Study 
Research  Design  (as  described  by  Yin,  2003) 

1.  Research  questions 


2.  Propositions 


3.  Unit  of  analysis 


Methodology 


4«r 


Component  #1:  Research  Questions 


•  rq  i:  What  is  the  present  model  used 
to  provide  Air  Force  Aerospace  Medical 
Technician  skills  sustainment  training? 


4.  Logic  linking  the  data  to  propositions 

5.  Criteria  for  interpreting  the  data 


•  IQ  1 :  What  evidence  suggests  a  formal  model  for 
aerospace  medical  technician  skills  sustainment 
training? 


9 


10 


w 


Improvised  Model 


m 

4UVT 


w 


Research  Questions  (continued) 


9 

- -  4UVFT 

RQ  2:  Have  there  been  recent  changes  to 

the  aerospace  medical  skills  sustainment 
training  environment  that  may  contribute  to 
the  success  or  failure  of  the  current 
model? 

•  IQ  2:  What  changes  have  occurred  with  respect 
to  land,  labor,  capital,  and  the  nature  of  work? 

•  Personnel  Reductions,  Loss  of  MTFs,  Declining  Skill 
Sets  for  assigned  personnel  (Inpatient,  ESD) 
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AFMS  MTF  Inventory 


4W 


140 
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Research  Questions  (continued) 
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•  RQ  3:  What  indicators  might  be  suggestive  of 
problems  with  the  current  training  model? 


•  IQ  3:  What  insights  can  be  gained  from  Air  Force 
Lessons  Learned  with  respect  to  Aerospace  Medical 
Technician  Skills  Sustainment  Training? 


16 
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S/  Quality  Indicators 


•  Training  Accessibility 


W 


Content  Analysis 


* 


•  59  Applicable  Lessons  Learned  Evaluated  Through 
Content  Analysis 


•  Training  Effectiveness 


•  1 00%  Reflected  Negative  Findings  With  Respect  To 
Current  Aerospace  Medical  Technician  Skills 
Sustainment  Training. 


•  (Bringslimark,  2004) 


•  Virtually  all  addressed  training  effectiveness  and/or 
training  availability  issues. 


17 


18 


w 


•  RQ  4:  Is  a  new  Aerospace  Medical  Technician  skills 
sustainment  training  model  warranted? 


W  Do  we  need  a  new  model? 


•  IQ  4:  What  are  some  of  the  existing  problem  areas 
the  new  model  should  address? 


•  Should  address  versatility  issues 

•  Effectiveness  issues 

•  Accessibility  issues 

•  Fidelity  Issues  (Realism) 


19 
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Sy  Versatility 


Deployed  vs.  Home-station 


“5”  Skill  Level  ANOVA 


•  Must  apply  to  both  deployed  and  Home-station 
training  environments 


•  There  is  a  statistically  significant  difference 


ANOVA:  Shglj  Factor 


SUMMARY 


i?rcnj|us 

Courrt 

Sum 

Ayxzgc 

Variant 

DEP  AD  0 

371 

8702 

23.61720 

200.3072 

HSAC5 

3T1 

7495 

20 .202'  6 

233.1617 

ANOVA 

Scoree  cf  Variation 

SS 

Df 

MS 

F 

P-va.‘je 

Fcfi 

Between  Groups 

WHh  n  Croups 

2163.402264 

1B99E3.4679 

1 

740 

2' 63.462 
256.7344 

8.426849 

0.003807 

3.854056 

Fatal 

192146.9501 

741 
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y  Deployed  vs.  Home-station 

-  JUVFT 

“7”  Level  ANOVA 


Anova:  Single  Factor 


SUMMARY 


Groups 

Count 

Sum 

/lVS,3p6 

Variance 

Dep  AD  7 

371 

9418 

25.38544 

368.4257 

1  IS  AD  7 

371 

6722 

18.1186 

158.C67 

ANOVA 

Source  cf  Variation 

SS 

Df 

MS 

F 

P-value 

Fcrit 

Between  Groups 
Within  Groups 

9795.708895 
195502  6631 

1 

710 

9795.709 

258.2468 

3651752 

2.4E-09 

3.854056 

Total 

206298,372 

741 
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Methodology  (cont.) 


Component  #2:  Proposition 


•  Study  Proposition:  Numerous  elements  of  the 
Aerospace  Medical  Technician  training  environment 
have  changed. 


•  What  must  change  with  respect  to  current  medical  skills 
sustainment  training  if  we  hope  to  maintain  a  high 
degree  of  medical  readiness  for  present  and  future 
Aerospace  Medical  Technicians? 

•Training  must  become  more  versatile,  accessible,  effective,  and 
realistic. 

24 


79 


Proceedings  of  the  2011  AFMS  Medical  Research  Symposium 

Volume  4  Healthcare  Informatics 


Methodology  (cont.) 


Component  #3:  Unit  of  analysis 


•  The  unit  of  analysis  is  the  Aerospace  Medical 
Technician  Skills  Sustainment  Training  Environment. 

•  Study  included  historical  data  analysis  (SGN),  content 
analysis  of  Lessons  Learned,  and  quantitative  analysis 
of  survey  data  (AFOMS) 


Methodology  (cont.) 


Component  #4:  Logic  linking  the  data  to 
propositions  (data  analysis) 

•  Pattern-matching  (Content  Analysis) 


•  Inductive  Logic  links  formed  through 
historical  analysis,  content  analysis,  and 
quantitative  data  analysis  (ANOVA) 


Methodology  (cont.) 


w 


How  Can  IT  Help? 


Component  #5:  Criteria  for  interpreting  the  data 


Networked  Learning 
Environment  aka.  ALN 


■ 


•  Do  the  results  support  the  proposition? 

•  AFOMS  data  supported  proposition  through  descriptive 
statistics  and  multiple  applications  of  ANOVA. 

•  Propositions/Findings  also  supported  by  existing  theory 

•  Knowledge  Management  Theory 

•  Training  Effectiveness 

•  Accessibility 

•  Organizational  Behavior  Theory 

27 
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^^High-Fidelity  Human  Patient  Simulator!^ 


Barriers  to  distance  based 
simulation  training  can  be 
minimized  through  the  integration 
of  remote  controlled  high-fidelity 
patient  simulators  with  an 
interactive  3D  video 
teleconferencing  /  telepointing 
internet  based  network,  whose 
geographical  range  is  unlimited. 
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Design  Quality  Criteria 


'M 

a arr 


Tesla 

Case  Study  Tic&o 

Pliase  of  research  .n 
which  tactic  ocai:s 

(.'onsrul  v  «l  idly 

•  Lse  tru  liple  sources  ol'cv  dencc 

•  Establish  chain  of  evidence 

•  Have  tcynlirnraiUs  review  draft  case 

study  report 

data  col  action 

dali  t-oi.wlion 

Internal  voLdity 

•  Dopnllem-matchirg 

•  1  \i  exp  ;m  ali  oi>  buildup 
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•  Use  replication  logic  in  niultiple^use 

studies 

lesetncli  ilesiwn 

research  design 

Reliability 

•  Use  case  study  protocol 

•  Develop  case  study  database 

dati  collection 

dati  collection 

Research  Benefits 


ra 

Am 


This  research  provides: 

•  A  broad-brush  view  of  problem  areas  regarding  required  training 
as  mandated  in  the  CFETP. 

•  Confirmation  of  decreasing  resources  experienced  by  Air  Force 
Aerospace  Medical  Technician  trainees  and  trainers. 

•  Confirmation  of  decreasing  experience  levels  exhibited  by  Air 
Force  Aerospace  Medical  Technician  trainers. 

•  Potential  application  of  findings  to  other  medical  related  AFSCs. 

•  A  potential  alternate  training  model  that  further  exploits  the 
advantages  of  existing  resources  such  as  IT. 

•  A  starting  point  for  future  research  to  test  model  efficiency. 

•  Insight  regarding  Air  Force  Aerospace  Medical  Technician  training 
issues/problems  (as  reported  by  AF  trainees  and  trainers). 


33 


34 


Limitations 


Am 


w 


•  Self-reported  bias  (surveys) 

•  Researcher  bias 

•  Immaturity  of  the  research  area/lack  of  fundamental 
literature 

•  No  test  of  proposed  model  effectiveness 
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Questions? 
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Special  Needs  Assessment  and  Planning  Environment  for  Emergency  Operations  Decision  Making 
Mr  Aaron  Miller,  711  HPW/USAFSAM-ETS 

During  man-made  or  natural  disasters,  significant  segments  of  the  population  have  special  medical  needs  that 
are  not  addressed  by  current  emergency  operations  processes.  These  patients  are  often  neglected  during 
disasters  due  to  limited  resources  resulting  from  insufficient  knowledge  of  a  system’s  capacity  to  respond  to 
their  needs. 

A  research  and  development  project  was  conducted  to:  (1)  assess  the  availability  of  detailed  infrastructure  data 
regarding  population,  medical  facilities,  and  transportation  resources  and  (2)  identify  a  simulation  tool  capable 
of  modeling  the  human  behaviors  of  victims  and  responders  during  an  emergency.  The  combination  of  these 
data  and  tool  resulted  in  the  Special  Needs  Assessment  and  Planning  (SNAP)  environment.  Modifications  to  the 
user  interface  include  app-based  access  from  mobile  devices.  SNAP  is  a  decision  support  tool  that  quickly  and 
easily  conducts  statistical  predictions  of  resource  needs  for  supporting  special  medical  needs  patients.  Further, 
as  actual  data  are  provided  to  the  Emergency  Operations  Center  (EOC),  reassessments  using  live  data  should 
provide  a  continuing  series  of  predictions  in  real  time. 

To  evaluate  the  fidelity  of  the  SNAP  environment,  SNAP  will  be  tested  in  Hamilton  County,  Ohio,  in  May 
during  “Shaken  Horizons,”  a  multiregion  exercise  simulating  local,  regional,  and  national  multidomain  response 
to  a  large-scale  earthquake.  SNAP  will  be  used  inside  the  county’s  Emergency  Planning  Collaborative  and 
provide  medical  facility  and  fire/rescue  resource  utilization  predictions  during  the  multiday  event.  Results  from 
the  actual  event  and  the  after-action  review  will  be  presented  to  illustrate  SNAP’s  utility  to  EOC  decision 
makers. 
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Special  Needs  Assessment  and  Planning 
(SNAP)  Environment 

Emergency  Operations  Decision  Making 
By  Aaron  Miller 
Wright  State  Research  Institute 
National  Center  for  Medical  Readiness 

Every  Airman  a  Force  Multiplier  3  AUQUSt  201 1 

August  2011  AFMS  Research  Symposium 
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The  Problem 


v  Significant  segments  of  the  v 
population  have  special 
needs  that  are  not 
addressed  by  current 
emergency  operations 
processes. 


Often  these  functional 
needs  populations  are 
neglected  due  to: 

Limited  resources 
resulting  from  insufficient 
knowledge  of  system 
capacity 

People  placed  in  shelters 
not  equipped  for  them 
v  Insufficient  planning  and 
preparation 

v  Inadequate  access  of 
medications 


Dtstitoullor  Slatemem  A:  Approved  tor  ptolic  release;  atstnotillor  is  urllmltea.  Case  Nunoer:  38A9W-201  *.-4092, 25  jul  201 1 


The  Problem 


Emergency  planners  and 
responders  need  to  know: 
v  How  many  people  need  to  be 
evacuated?  How  many  people 
will  need  medical  attention? 

v  How  many  neonatal  patients  will 
be  in  the  disaster  area? 
v  How  many  dialysis  patients  will 
require  treatment? 
v  What  equipment  is  needed  to 
respond  to  this  disaster  and  the 
associated  population? 


Where  should  the  temporary 
held  hospital  be  set  up  and  how 
should  I  organize  resupply? 


The  Solution:  SNAP  Environment 

v  SNAP  is  an  environment  to  facilitate  the  analysis  of  the 
emergency  response  to  a  man-made  or  natural  disaster. 


r  ■Wfr- 


Identify  at-risk  functional  needs  populations. 
v  Neonatal  patients/new  births 
v  Dialysis  patients 
v  Oxygen-dependent  patients 
v  Mental  health  patients 
v  Medically  dependent  elderly 
Support  resource  decisions. 
v  Personnel 
v  Equipment 
v  Consumables 
v  Transportation  or  evacuation 
Provide  relevant  data  earlier  in  a  disaster  to  enable  planning 
and  expedited  response. 


from  s  rehaaittasion  oents r  *o>lo«1r»g  Hunicarrs 
Katran 
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v 


v 


V 


The  Solution:  SNAP  Environment 


■> 


Leverages  existing  government  or 
academic  databases. 


Ties  data  together  using 
behavioral  models. 


Predicts  the  number  of  people  and 
their  expected  medical  conditions 
who  will  likely  need  evacuation 
and  medical  care. 


Otslrtoutton  Statement  A:  Approved  for  outlie  release:  dtstneutlon  is  unarmed  Case  Mumper  S-3ABW-2G1 1-1052. 25  Jul  2011 


Yhe  Solution:  SNAP  Environment 


*31 


v  Connects  functional  needs  care  and  evacuation 
requirements  with  critical  disaster  medical  resources. 
v  Allows  community  planners  to  predict  high-urgency 
resource  requests 
within  geographical 
context. 

v  Assists  emergency 
management  agencies 
with  ADA  Title  II 
compliance  for 
vulnerable  functional 
needs/disabled 
populations. 


Cisinoullon  Statement  A:  Approved  for  oucnc  release:  dlstmutlon  is  unamlted  Case  Mumper  58ABW-201 1-4092. 25  Jul  201 1 


The  Solution:  SNAP  Environment 


SNAP  Research  &  Development 


v  Provides  emergency 
operations  center  personnel 
with  a  decision  support  tool 
containing  data-validated, 
modeled  outcomes  both 
before  and  during  events. 
v  Allows  planners  to  tailor 
disaster  planning  needs  to 
locally  existing  databases. 
v  Visualizes  simulation  results 
for  easy  recognition  of 
significant  findings. 
v  Extremely  valuable  tool  for 
mass  gathering  and  large- 
scale  public  event  planning 


v  To  validate  the  capabilities  of  SNAP,  a 
research  and  development  project  was 
conducted  to: 

v  Identify  and  validate  a  simulation  tool  capable  of 
modeling  the  human  behaviors  of  victims  and 
responders  during  an  emergency 
v  Assess  the  availability  of  detailed  infrastructure  data 
regarding  population,  medical  facilities,  and 
transportation  resources 


Cisinoullon  statement  A:  Approved  for  punilc  release,  distnoullon  is  unarmed 
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Research  Activities 


Real-Time  "Decision  "Support 


v  Conduct  research  to  determine  viability  of  using 
SNAP  as: 

v  Real-time  decision  support 
v  Planning  tool 
v  Exercise  tool 

-  Evaluate  viability  of  data  associated  with  SNAP 
environment  by: 

v  Identifying  renewable  accurate  data  sources  for 
inputs 

v  Determining  value  and  appropriate  level  of  complexity 
of  data  outputs 


v  While  SNAP  can  be  used  as  a  real-time  decision  support 
tool,  the  current  limitation  on  SNAP  is  the  lack  of 
integrated  real-time  data  sources. 
v  Currently  behavioral  models  represent  human  behavior, 
first  responder  actions,  and  facility  utilization. 
vAs  live  data  sources  become  available,  they  can 
replace  the  behavioral  models  to  provide  real-time 
data  enabling  integration  into  command  and 
control  (C2)  platforms. 

Real-time  capability  is  viable  through  periodic  updates  of 
data  sources  to  support  real-time  planning  during  a 
disaster. 

v  Update  model  with  current  information  as  it  becomes 
available  and  rerun  model  to  obtain  planning  information 
for  decision  makers  ( e.g .,  represent  a  hospital  collapse). 


Cistrtoutlor  Statement  A:  Approved  (Or  pueoc  release:  distnpLllon  is  unlmlted  Case  Sumner  ESABW-201  1-1052. 25  Jul  2Gi  1 
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ra 

Incident  Planning  Tool 

' 

v  Provides  Emergency  Operations 

Center  managers  with  data- 
vaii dated,  modeled  outcomes. 

v  Allows  planners  to  tailor  disaster 
planning  needs  to  locally  existing 
databases. 

v  Visualizes  simulation  results  for 
easy  recognition  of  significant 
findings. 

v  Extremely  valuable  tool  for  mass  y 

gathering  and  large-scale  public  event  a 

planning  > 

Enables  the  identification  of  break 
points  within  the  emergency 
response  system. 

Allows  community  planners  to 
predict  high-urgency  resource 
requests  within  geographical 
context. 

Exercise  Tool  Mx 

'  t 

Extremely  viable  tool  for  use  in  exercise  planning 
and  training  for  disaster  exercises  and  events 
v  Generates  injects  for  more  realistic  scenario  exercises. 
v  Generates  holistic  scenarios  with  integration  between 
disparate  entities. 

v  Serves  as  command  and  control  module  for  exercise  to 
determine  when  activities  should  be  concluded  and 
kicked-off  during  exercise  execution. 
v  Allows  for  management  of  virtual  and  constructive  resources 
within  a  live  exercise. 

v  Can  recreate  exact  scenario  to  reinforce  training 
activities. 

v  Is  capable  of  integration  into  live,  virtual,  and 
constructive  training  activities. 

Distnoullon  statement  A:  An  proves  tor  puHlc  release,  otstnoutlon  Is  unimiced  Case  Sumner:  S3ABW-201 1-4052, 25  Jul  2011 
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Data  Viability 

The  SNAP  environment  is  composed  of  a  large  number  of  data  sets  required  to 
obtain  the  appropriate  fidelity;  data  sets  represent: 

Functional  need  populations 

Infrastructure 

-  Fire  fi.e.,  ambulances,  fire  apparatus) 

Medical  (i.e.,  hospitals,  dialysis  centers) 

Civic  (i.e.,  public  works,  shelters,  etc.) 

Personnel  resources 

Renewable  data  sets  are  required  to  simplify  maintenance  and  expandability  of 
the  data. 

-  Epidemiological  research  found  significant  pools  of  renewable  data  sets 
viable  to  support  the  SNAP  environment. 

Functional  needs  patients  for  the  entire  country  with  the 
exception  of  oxygen  dependents 

Medical  infrastructure  on  state- by -state  basis  (variable  formats) 

Civic  infrastructure  on  state -by -state  basis  (variable  formats ) 

Data  Viability 

v  The  inconsistency  between  state  data  sets  does  provide 
challenges  in  integrating  the  solutions  into  the  SNAP 
environment. 

v  Inconsistent  data  definitions 

v  Inconsistent  data  formats  and  data  values 

v  General  cause  is  a  lack  of  national  standards  in  recording 
and  reporting  key  data  elements  of  interest  to  disaster  and 
emergency  first  responders 

v  Challenges  overcome  through  the  use  of  XML  and 
extraction,  transformation,  and  load  (ETL)  procedures 
v  Other  elements  of  fidelity  within  the  data  (i.e.,  increase  in  birth 
rates  during  a  disaster)  have  been  enabled  as  editable  fields  in 
the  user  interface  to  allow  for  updates  as  necessary. 

CtstirouBor  Slats  mem  A:  Approved  for  puollc  release;  ®strioul)on  is  uniimneo.  Case  Mumper  SSA3W-20ii-i092. 25  Jul  20ii 

OistitDullor  Statement  A:  Approved  for  puollc  release;  oistitoutlon  is  urnmltea.  Case  Mumper:  S8A3W-201 1-1092. 25  Jul  201 1 

"Summary 


Fidelity 


Currently,  SNAP  is  best  used  as  a  planning  and  exercise  tool  for 
evaluating  impacts  of  disasters  on  resources  and  functional 
needs  populations. 

v  Realistic  and  accurate  representations  of  exercise  injects 
v  Serves  as  C2  capability  for  training  activities 
v  Performs  break  point  analysis  of  key  system  infrastructure  and 
resources 

SNAP  is  capable  of  serving  as  a  decision  support  tool  for  first 
responders. 

v  Improved  integration  to  real-time  databases  would  dramatically 
enhance  real-time  capabilities 

Existing  data  are  sufficient  to  support  the  SNAP  environment; 
however,  standardization  in  the  reporting  of  data  at  the  state  level 
would  greatly  enhance  the  usability  of  data. 


Functionality  of  model  is  based  on  behavioral  models. 
v  How  people  actually  behave  during  a  disaster,  so  it  is 
accurately  represented  in  the  simulation 

a  Examples: 

■  Disease  and  prophylactic  models 

Disease  state  tracking  of  population 
Prophylactic  models 

■  Medic  ine  efficacy  and  effect  on  population 

■  Epidemiological  models 

■  Logistics  models 

Medicine  distribution  logistics  modeling 

■  Medicine  consumption 

■  Distribution  center  medicine  stock  depletion  and  replenishment 
•  Communication  between  centers 

Additional  Features 


OetffiXiUon  statemert  A  Approved  for  puollc  release;  astflpullon  la  urnmltea  Cass  Mumper.  3SA3W-201 1-40B2. 25  Jul  2Q11 
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"Fidelity:  An  "Example 


SNAP  Inputs 


During  disasters,  there  is  an  increase  in 
birth  rate  due  to  stress  placed  on  mothers, 
creating  a  higher  level  of  premature  births. 


v  Two  types  of  input: 
v  Specific  entity  data 

Population  units  (any  geographic  area  were  data  are  available) 
v  Medical  facilities  (nursing,  hospitals,  dialysis  centers) 
Resources  such  as  fire,  rescue,  public  works,  and  buses 
v  Run  time  user  options 

s  Disaster  mechanism  and  damage  likelihood  control 
v  Medical  data 

•  Patient  timelines 

•  Initial  conditions  at  medical  facilities 

v  Parameters  to  control  tactics,  techniques,  and 
procedures 


Ctstrtoullon  Stale  mem  A:  Approved  for  puollc  release;  sstflputlon  is  unlimited.  Case  NumDer.3SA3W-20il-iOS2.2S  Jul  20il 
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Control  Tour  "Simulation 


•-v 


"Modeling  Overview 


Medical  Facilities 


v  User  has  substantial  control  to  modify  default 
values  critical  to  simulation  outcome. 
v  Assets  and  timeline  of  “external”  responders 
v  Disaster  damage  likelihood 
v  Medical  need  timeline  for  each  patient  type 
v  Treatment  times 

Initial  conditions  (facility  fullness,  emergency  rates) 
v  Evacuation  parameters  (transportation  needs) 
v  Shift  duration  for  fire/rescue  resources 
v  Power  restoration  timeline 
v  Average  transport  speed 


v  Capacity  to  accept  new 
patients 

v  Treatment  of  current  patients 
v  Fire/Ambulance  Resources 
v  Response  to  emergency 
calls 

v  Transport  patients,  if 
applicable,  to  medical  facility 

v  Patients 

v  Many  are  self-moving  and 
search  out  facilities  to  fill  their 
specific  medical  needs 
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Study  Matrix 


v  Using  a  study  matrix,  users  can  generate  and 
execute  studies  to  determine  breaking  points  for 
resources. 

v  Patient  fatalities  and  treatment  needs  as  a  function  of 
size  and  scope  of  disaster 
v  Impact  of  nominal  speeds 
v  Communication  delays 
v  Power  restoration  rate 
v  Loss  of  specific  facilities  or  resources 
v  Number  of  re-routes  as  a  function  of  initial  capacity  of 
medical  facilities 

v  Transportation  or  shelter  resources  required 
v  Likelihood  of  medical  facility  capacity  issues 
v  Sensitivity  of  damage  likelihood 


DWimunon  Slalemert  A:  Aporoved  for  plzmic  re=«asa;  ostdOLtlor  is  urlinlteo.  Case  Nunoer:  38A5W-20ii-40S:,  25  Jul  2D11 
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You  Won't  Get  There  From  Here  Without  Getting  Them  Here-- AFMS  Diabetes  Care  Quality 
Measurement 

Ms  Brooke  Asbury,  Office  of  the  Chief  Information  Officer,  Air  Force  Medical  Support  Agency 

Initial  work  by  the  Air  Force  Medical  Service  (AFMS)  Applied  Clinical  Epidemiology  (ACE)  team  quantified 
the  AFMS  diabetic  “enrolled  but  not  seen”  population  to  be  13%  of  the  overall  diabetic  population.  HEDIS® 
scores  for  the  “not  seen”  population,  defined  here  as  diabetics  having  no  MTF  outpatient  encounters  in  one 
year,  are  much  lower  than  scores  for  those  seeking  MTF  care.  To  further  characterize  diabetics  “not  seen,” 
ACE  examined  enrollment  status,  MTF  characteristics,  referral  histories,  and  encounter/medication  histories 
using  AFMS  clinical  informatics  data. 

Seventy-five  percent  of  diabetics  “not  seen”  had  TRICARE  Plus  or  other  health  insurance  vs.  thirty-six  percent 
of  “seen”  diabetics.  Seventy  percent  of  “not  seen”  diabetics  had  two  or  more  billed  network  outpatient 
encounters  for  diabetes  or  had  at  least  one  diabetic  network  encounter  plus  medication(s).  Similar  proportions 
of  those  “seen”  and  “not  seen”  had  emergency  department  visits,  inpatient  stays,  and  diabetes  medication(s). 
Among  those  “not  seen,”  few  referrals  to  the  network  (<2%)  existed,  and  fewer  MTF  teleconsults  were  found 
vs.  were  found  for  diabetics  “seen.” 

Diabetics  “not  seen”  obviously  have  proportionally  higher  utilization  of  and  access  to  non-MTF  care  compared 
to  “seen”  diabetics  based  on  enrollment  status,  low  volume  of  referrals  to  network,  and  the  high  percentage 
having  network  bills.  Most  are  obtaining  diabetes  care,  though  the  quality  of  care  received  in  the  network  is 
unclear.  Across  MTFs,  HEDIS®  scores  trend  upward  as  percentages  of  “not  seen”  and  TRICARE  Plus 
diabetics  trend  downward.  Systematic  efforts  targeted  to  diabetics  ’’not  seen"  are  necessary  to  positively  impact 
AFMS  HEDIS®  scores. 
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Headquarters  U.S.  Air  Force 

Integrity  -  Service  -Excellence 


AFMS  Diabetes  Care  Quality 
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Integrity  -  Service  -  Excellence 


U.S.  AIRFORCE 


Why  we  measure  quality 


How  we  measure  and  report 

quality 


■  It  is  required  by  law 

■  Section  723(e)  of  the  National  Defense  Authorization  Act  for 
Fiscal  Year  2000,  Public  Law  106-65 

■  From  the  Department  of  Defense  Health  Care  Quality  Report  to 
Congress,  2010: 

■  “77ie  MHS  is  committed  to  being  patient  centered  and 
providing  quality  health  care  ”  (p.  4) 

■  “ On  and  off  the  battlefield \  in  times  of  peace  and  war,  the 
MHS’s  goal  is  to  ensure  that  the  highest  standard  of  care  is 
delivered.  ”  (p,  1) 

■  “...to  enhance  the  quality  of  care  provided  at 

MTFs...  payments  for  quality  of  clinical  care  are  based  on 
perfotmance  on  HEDIS®  and  ORYX®  measures/'  (p.  xi) 
*HEDIS^  Healthcare  Effectiveness  Data  and  Information  Set® 


■  Air  Force  Medical  Support  Agency  (AFMSA/SG6H)  produces 
selected  clinical  quality  measures  for  DoD 

■  Measures  are  refreshed  monthly  on  MHS  Population  Health 
Portal  (MHSPHP) 

■  Action  lists  are  provided  for  population  health  management 

■  Measures  and  lists  are  produced  using  HEDIS®  methodologies 

■  Include  TRICARE  Prime  and  Plus  enrollees  to  Military 
Treatment  Facilities  (MTFs) 

■  Use  Administrative  Specification  only 

■  Include  several  diabetes  care  measures 

■  DoD  uses  the  National  Committee  for  Quality  Assurance’s 
(NCQA)  Commercial  HMO  benchmarks  for  comparison 


Integrity  -  Service  -  Excellence 


Integrity  -  Service  -  Excellenc 
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Diabetes  care  quality  measures 


U.S.  AIR  FORCE 


Current  performance — 
„^c.  HbAlc  Screening  (Direct  Care) 


■  Diabetics  are  identified  through  encounters  and  medications 

■  Lists  are  further  refined  using  HEDIS®  exclusion  criteria 

■  Percentages  of  diabetics  screened  and  “in  control”  are  presented 

■  Four  measures  for  Hemoglobin  A1C  (HbAlc) 
screening/control 

■  HbAlc  screening  in  past  year 

■  HbAlc  <=  9%,  HbAlc  <8%,  HbAlc  <7%  (diabetics  without 
selected  comorbidities) 

■  Two  measures  for  Low  Density  Lipoprotein  (LDL) 
screening/control 

■  LDL  screening  in  past  year 

■  LDL  control  (<100  mg/dL) 


Integrity  -  Service  -  Excellence 


Direct  Care:  Diabetes  Hb  A1C  Screenings  Ft  2005-2009 
160  .  , 


Source:  DoD  Health  Care  Quality  Report  to  Congress, 
2010 


Integrity  -  Service  -  Excellence 


Current  performance — 
JL- _ HbAlc  Screening  (USAF)  „ 


Current  performance — 
HbAlc  <=  9%  l USAF ) 


Air  Force  -  Diabetes  HgAIC  Screening  -  HEOIS  -  (7/24/2011) 
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MHS  Pop 

Health  Portal  2010  Benchmarks:  90th  -  93,7%,  50th  -  89.0%  |  2011  Benchmarks: 

90th  -  93.7%,  50th  -  89.0% 
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Air  Force  -  Diabetes  HgAIC  <=9  Control  -  HEDIS  - 
(7/24/2011) 
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2010  2010  2010  2010  20102011  2011  2011  2011  2011 


Health  Portal  2010  Benchmarks:  90th  -  81. 3%,  50th  -  72.2%  |  2011  Benchmarks: 
90th  -  81.3%,  50th  -  72.2% 
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Integrity  -  Service  -  Excellence 
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mS.  AIR  FORCE 


Current  performance — 
HbAlc  <  8%  (USAF) 


U. S.  AIR  FORCE 


Current  performance — 
HbAlc  <  7%  (USAF) 


Air  Force  -  Diabetes  HgAIC  <8  Good  Control  -  HEDIS  - 
(7/24/2011) 
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Air  Force  -  Diabetes  HgAIC  <7  Good  Ctrl  no  Comorto  - 
HEDIS -(7/24/2011) 


Integrity  -  Service  -  Excellence 


Source: 

MHS  Pop 

Health  Portal  2010  Benchmarks:  90th  -  54.3%,  50th  -  43.7%  |  2011  Benchmarks: 
90th  -  54.3%,  50th  -  43.7% 

Integrity -  Service-Excellence 


Current  performance — 
J:Lc. LDL  Screening  [ USAF 9 

Air  Force  -  Diabetes  LDL  Screening  -  HEDIS  -  (7/24/201 1 ) 
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Health  Portal  2010  Benchmarks:  90th  ■  B9.8%,  50th  -  85  1%  |  2011  Benchmarks 
90th  -  B9.B%,  50th  -  85.1% 


Current  performance — 
LDL  <  100  mg/dL  (USAF) 


Air  Force  -  Diabetes  LDL  Control  -  HEDIS  -  (7/24/2011) 


Source: 

MHS  Pop 
Health  Portal 


Integrity  -  Service  -  Excellence 


Integrity  -  S  e  r  v  i  c  e  -  E  x  c  e  1 1  e  n  c  e 
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U.S.  AIR  FORCE 


Why  are  we  not  meeting 
targets? 


■  USAF  is  not  reaching  50th  percentile  for  screening  measures 

■  Performing  somewhat  better  on  control  measures 

■  Possible  reasons  for  less  than  optimal  performance 

■  Appropriate  care  is  not  being  delivered  at  recommended 
intervals  by  MTF  providers 

■  Diabetics  are  not  compliant  with  diabetes  care  plan 

■  DoD  does  not  use  Hybrid  Specification  for  HEDIS0  measures 
(i.e.,  no  medical  record  review) 

■  Administrative  healthcare  data  for  selected  diabetics  is  not 
making  it  into  DoD  repositories 
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Taking  a  closer  look 


■  Air  Force  Medical  Operations  Agency  (AFMOA)  Applied  Clinical 
Epidemiology  (ACE)*  team  was  created  in  2010 

■  Has  medical  oversight  and  direction  of  AFMOA 

■  Has  analytical  expertise  and  informatics  resources  of  AFMSA 

■  Issue  of  diabetes  care  quality  met  ACE  investigative  criteria 

■  Important  to  leaders 

■  Measurable 

■  Data  regarding  diabetes  care  are  obtainable 

■  Presumably  can  be  impacted  by  patient  and  provider  actions 

■  Initial  analysis  stratified  HbAlc  and  LDL  outcomes  for  March 
2010  AF  MTF-enrolled  diabetics  by  past  year’s  visit  histories 

*ln  summer  2011 ,  ACE  was  renamed  “AF  Clinical  Decision  Support”  (AF  CDS) 
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Encounter  histories  shed  some 


-HE  DIS  delinilon  diabetes  as  of  31  Mar  201 0  {minus  continuous  eniollmenl  criterion ;  >1 3  yrs  of  ago) 

-Oulpalienlenoounters  tern  Apr  03-  Mar  10  {12  months) 

-Hi  A1  clLD  L  resull  on  record  {CHCS)  0  R  purchased  Care  "setvice  dale'*  no  dala  puls  palenl  in  'Viol  con  tolled"  category  Py  nJe 
-"HE  Dislike"  because  Ihe  upper  age  lirrtl  of  75  and  conlnuous  erroilmenl  ciilerion  werenol  applied 
source:  AFM5  AASC6H  Sepi  201 0 
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Does  “enrolled  but  not  seen” 
population  vary  across  USAF? 


♦  ♦  ♦♦ 

*  •  ♦  -  «  A  ♦♦ 

i 

»♦  ♦  ♦  ♦*  ♦  ♦  ♦  *  ♦  +  s  +  ♦ 

V *  ♦ 

1 

III  INI  iiiiii.. 

■MM 

lllllll  lllllllliiiiiiiiiiiiiiiiiiniii . . . 

„  II  n  i  l  h  i  i.i  i.i  i.i  i  i.i  i.i  m  i  M  m  i  1 1  h  i  i  i  m  i  1 1  i  l  l  1 1  i  l  l  1 1  in  i  i  i  i  i  .  . .  . . .  . . . 


m%  of  scion  list  wifi  no  Dired  ouVBfenU  visils  <Ma«sh  20 10  >  ©HbAlc  aranhg 
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Are  they  being  referred 
out  for  specialty  care? 


“Seen”  vs.  “Not  Seen” 

■  Similar  proportions  had  % 
diabetes-related  ER 
visits  in  past  year 

60 

■  Similar  proportions  had 
diabetes-related 
inpatient  stays  in  past  * 
year 


■  Similar  proportions 
were  on  diabetes 
medication 


2Q 


Had  dabele^ooded  ER  Had  diabetes-codad  Had  diabetes  meds 


visits}  irpsltertl  slay{s}  dspenaed 

■  Have  DC  OP  faoe-to-feoe  Mails  □  Have  na  DC  OP  faoe-te-face  visits 


“Seen”  vs.  “Not  Seen” 

■  “Seen”  diabetics 
were  much  more 
likely  to  have 
referral  records  in 
the  Referral 
Management 
System 


1QQ 

90 

33 


Had  any  referral  recod  of  aiy  s  tel  us  Had  "Deter  te  Network*  dr  "Defer- 
wte  a  referral  date  in  pasl  >ea-  Results  Reoerved"  slalus  wflh  retertf 
date  in  past  year 


D  Have  DC  o  P  face-to-teoe  visits  □  Have  no  DC  OP  Tace-te-feoe  visits 
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Are  they  obtaining  care 
elsewhere? 


W  Are  they  different 

demographically? 


“Seen”  vs.  “Not  Seen” 

“Not  seen”  twice  as 
likely  to: 

■  Have  other  medical 
insurance  (OHI) 

■  Have  TRICARE  Plus 
and/or  be  age  >=  65 


ina*«nce  (ACV-<3,L>  TRICARE  Plus  Plus  but  Have 

based  on  {ACV-G.L}  OHI(Dec2010 

DEERS ''OHI  He) 

Medical"  flag 
(Dec  2010  lie) 


■  Have  DCOP  faoe-to-feoe  Mans  □  Have  no  DC  OP  fece-to-Taoe  Mails 
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What’s  different  about  Plus? 
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Impact — LDL  screening 


■  TRICARE  Plus  vs.  TRICARE  Prime 
■  TRICARE  Plus 

■  Not  a  health  plan 

■  An  “enrollment  option”  for  Standard  enrollees  in  orderto 
receive  same  access  standards  as  TRICARE  Prime 

■  Plus  enrollees  are  assigned  a  Primary  Care  Manager  (PCM) 

■  Mainly  beneficiaries  age  65  and  over 

■  Access  to  MTF  specialty  care  is  not  guaranteed 

■  MTF  not  required  to  offer  Plus 


From  TRICARE.mil  website: 

http  ://www.tr  icare  .mil/mybe  nef  it/h  ome/ove  rview/S  pec  ia  IP  ro  g  rams/P  I  us 

Integrity  -  Service  -  Excellence 


U.S.  AIR  FORCE 


What  can  be  done? 


■  Acknowledge  the  impact  of  the  “not  seen”  group  on  quality 
measures 

■  Continue  to  provide  care  as  usual 

■  Enter  outside  data  into  medical  record  when  available 

■  Bring  all  diabetics  in  fora  visit  with  PCM  each  year  and  draw  labs 

■  Monitor  proportion  “not  seen”  and  their  outcomes  at  Patient- 
Centered  Medical  Home  MTFs  to  see  if  PCMH  makes  a  difference 

■  Internally  adjust  by  MTF  proportion  of  “enrolled  but  not  seen” 

■  Calculate  lower  “achievable”  benchmarks 

■  Remove  TRI CARE  Plus/OHI  from  denominators/numerators 

■  Politely  ask  the  “not  seen”  to  disenroll  from  MTF  (?)  if  they  are 
not  seeking  direct  care 
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Remaining  challenges 
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Conclusion 


Complex  policy  issue  for  both  DoD  and  MTF  leadership 

■  Difficult  to  determine  how  closely  measures  reflect  actual 
clinical  quality  and  how  much  importance  to  place  on  them 

■  MTFs  are  resourced  based  on  enrollment 
AFMSA/SG6H  undergoing  first  HEDIS®  compliance  audit 

Still  no  requirement  for  network  lab  results  to  enter  DoD 
repositories 

Difficult  to  make  blanket  decisions  about  “enrolled  but  not  seen” 
population,  as  it  is  a  somewhat  heterogeneous  group  of  enrol  lees 


■  “Enrolled  but  not  seen”  diabetics  will  continue  to  prevent  AFMS 
from  reaching  targets  for  diabetes  care  quality  measures  unless 
changes  are  made  at  multiple  levels 

■  Programs  that  use  quality  measures  downstream  of  MHSPHP, 
like  pay-for-performance  initiatives,  are  also  impacted 

■  Policy  changes  most  effective  if  enacted  at  DoD  level 

■  Even  after  “not  seen”  were  removed  from  March  2010  diabetes 
care  HEDIS®  measures,  HbAlc  screening  remained  under  90th 
percentile  compared  to  Commercial  HMOs 

■  Still  work  to  do  among  enrollees  that  visit  direct  care 

■  USAF  rates  have  been  flat  for  years 

■  In  the  meantime,  MTFs  can  use  MHSPHP  to  assist  with  patient 
identification,  disease  management  and  quality  assessment 
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Contact  Info 


BROOKE  L  ASBURY,  MPH 

AFMS  Office  of  the  CIO  (AFMSA/SG6H) 

Sr  Informaticist 

3515  S  General  McMullen  Suite  200 
San  Antonio,  Texas  78226 
COMM  210.395.9732 
DSN  969.9782 

Brooke.asbury.ctr@us.af.mil 
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A  Simulation-Based  Program  to  Improve  Non-technical  Skills  during  Cardiopulmonary  Resuscitation 
CPT  Albert  Bonafacio,  USAR,  Patient  Safety  Center  of  Inquiry,  Durham  (NC)  VAMC 

Introduction:  Sudden  cardiac  arrest  is  the  leading  cause  of  death  in  the  United  States.  Code  response  training 
has  traditionally  focused  on  improving  individual  responders’  technical  skills  and  knowledge  base.  However, 
the  impact  of  code  response  team  performance,  blending  interpersonal  and  cognitive  skills  (“non-technical 
skills”),  is  increasingly  recognized  as  critical  for  success  in  these  scenarios.  Since  these  skills  are  rarely 
evaluated,  we  developed  a  program  for  training  and  evaluating  non-technical  skills  in  code  scenarios. 

Methods:  A  high-fidelity,  simulation-based  program  to  improve  non-technical  skills  among  in-hospital  code 
responders  was  implemented  at  a  tertiary  VA  Medical  Center.  The  Cardiopulmonary  Resuscitation  Team  (CRT) 
program,  comprised  of  three  components  (education,  program  evaluation,  and  quality  improvement),  was 
introduced  to  rotating  departmental  house  staff  over  one  year.  Participants  were  oriented  to  code  roles  and 
responsibilities.  Six  times/month,  8-minute  simulated  arrest  scenarios  were  conducted,  followed  by  debriefing 
emphasizing  communication/teamwork.  Simulated  code  scenarios  were  videotaped  and  reviewed  to  evaluate 
CRT  performance  with  respect  to  non-technical  skills. 

Results:  Simulated  code  exercises  were  significantly  improved  with  regard  to  task  performance, 
communication,  and  organization,  which  has  translated  to  more  efficient  “real-world”  codes.  Numerous  parallel 
processes  relevant  to  CRT  performance  (code  cart  organization,  modified  acquisition/delivery  of  laboratory 
samples,  code  documentation)  have  been  improved  and  applied  to  actual  clinical  events. 

Conclusions:  Non-technical  skills  are  essential  to  successful  resuscitation  efforts.  The  CRT  program  used  high- 
fidelity  simulation  to  enhance  and  maintain  non-technical  skills  among  in-hospital  cardiac  arrest  responders. 
Comparison  of  pre-  and  post-implementation  in-hospital  cardiac  arrest  mortality  data  will  be  evaluated  to 
further  assess  program  effectiveness 


[No  Presentation  Slides  Follow} 
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Utilization  of  a  Prescreening  Instrument  for  the  Selection  of  Special  Duty  Personnel 
Dr.  Joe  Wood,  III,  711  HPW/USAFSAM-FEC  WPAFB  OH 

Selecting  the  highest  caliber  personnel  for  Air  Force  special  duty  assignment  is  crucial  for  reducing  training 
attrition,  increasing  retention,  and  improving  operations  critical  to  national  security  mission  readiness  and 
completion.  The  procedures  for  assessment  and  selection  of  special  duty  personnel  can  be  a  time-consuming 
and  expensive  process.  However,  utilizing  an  empirically  validated  prescreening  instrument  can  be  one  of  the 
more  cost-effective  methods  of  refining  the  applicant  pool  prior  to  an  in-person  assessment  and  selection 
(A&S),  thus  avoiding  the  costs  associated  with  travel,  lodging,  and  lost  time  on  the  job  for  the  applicant  in 
addition  to  reducing  the  resources  needed  by  staff  at  the  A&S. 

This  study  evaluated  the  usefulness  of  an  empirically  validated  ’’select  out”  web-based  prescreening  instrument 
assessing  medical,  psychological,  and  interpersonal  aspects  of  functioning.  Out  of  the  1100+  potential 
applicants  who  completed  the  prescreen  survey  between  2005  and  2009,  approximately  52%  were  identified  as 
having  concerning  information  affecting  their  fitness  and  suitability  for  a  high-demand,  high-risk  special  duty 
career  field.  In  total,  78%  of  those  flagged  were  eliminated  from  consideration  after  additional  review  by  unit 
leadership.  These  eliminations  are  estimated  to  have  provided  savings  of  more  than  $200,000  per  year. 
Additionally,  the  use  of  the  instrument  has  significantly  improved  (a)  the  quality  of  the  pool  of  applicants 
invited  to  attend  A&S  and  (b)  our  understanding  of  the  prerequisites  needed  to  successfully  adapt  to  the  training 
and  operational  rigors  of  a  special  duty  assignment. 
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Overview 


Utilization  of  a  Prescreening 
Instrument  for  the  Selection  of 
Special  Duty  Personnel 


Joe  Wood  III,  PsyD 


Eagle  Applied  Sciences 
Aeromedical  Consultation  Service 


USAF  School  of  Aerospace  Medicine 


v  History  of  assessment  &  selection  (A&S) 
v  Description  of  special  duty  program 
v  Characteristics  of  screening  instrument 
w  Benefits/cost-savings  of  prescreen 


Distibuton  Statement  A.  ApfMued  T<*  fhblie  release.  distibulbi  is  uiliniled .  Case  Number.  S8ABW-33 1 1  -4 063. 25  JOI 20 11 


DisVibuldm  statement  A.  ^i^oved  Tor public, release,  distributee  is  uilimiled .  Case  Number. 8SABW-33 1 1 -4063. 25  Jul  20 1 1 
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History  of  A&S 


v  World  War  I 

v  Army  Alpha  &  Beta  tests 
v  Screened  1-2  million  soldiers 
v  Aptitude  test  for  job  placement 
v  Army  Alpha  &  Beta  tests 

The  most  prominent  industry  of  Minne  apolis  is: 
a.  Hour 

B.  Packing 

C.  Automobiles 

D.  Brewing 

It  is  better  to  fight  than  run  because: 

A.  Cowards  are  shot 

B.  It  is  more  honorable 

C.  If  you  run  you  may  get  shot  in  the  back 

Distributed  Statement  A.  fyfravea  piblie release.  distibulbl  is  ualirrited.  Case  Number.  38ABW-2011  -4063. 25  Jul  3311 
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History  of  AScS 


v  Office  o  f  Strategic  Services 
(OSS) 

v  Precursor  to  the  CIA 

v  Screened  for  intelligence  and 
training 

v  No  testing  for  emotional/social 
attributes 

v  Problems  with  field  agents  having 
nervous  breakdowns 


Disti  baton  Statement  A.  fyfraved  to- pallia  release.  distibulbt  is  iblimJled.  Case  Number.  SSABW-231t -4363, 25  Jul  3311  * 
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History  ofA&S 


v  Office  of  Strategic  Services  (OSS) 

v  pl  forma/  attempt  at  A&S  for  special  duty  personnel 
v  Selection  run  by  clinical  psychologists  &  psychiatrists 
Evaluated  5,391  recruits  from  1 943- 1947 


f*1  attempt  at  comprehensive  A&S 
■  Intelligence  tvstlng 
Psychological  testing 
v  Fiaid  testing 
>  Motivation 


.mjiiimw*  3+i. 
CaeN&r  ru:  rm 

AiU.  OSS  i.  iipWEfl]!  ■flrf'O 
1C  liTUJJSgfc: 


3— * 


swinii-4fflaiB  Jii 


vNASA  se!ect-in  criteria 

-  Ability 

tn  tniSgansa 
d-tfrtrfo/sflJp 
Sim  s.1!  ic-iwjtnc# 

Flexibility 

v  Stabs  lit y 

Hardlnesss 

Ability  to  form  stable  relationships 
Expressivity 
Assertiveness 

Sens?  of  hurrwr/euftuntl  sensitivity 
v  Motivation 

Maxtary 

Work  oriontafiien 
Campaiilivenaxx 
Extrlnsio  meth/a  tion 


DttufciiSm  Samara  i.  alv n**d  tel  mbit-  'anHUi  of sitfbuiM  In,  mlirM  Hi-ntor.  £1  Jd  M*i 
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Special  Duty  Program 


"  Structured  A&S  began  in  1993 

Prior :  Informal  interviews;  by-name  recommendation 
v  First  research  funds  in  2001  (5  phases  through  2010} 

V  GOALS: 

Validate  (fafmess/accuracyj-Does  It  work?  How  do  you 
know? 

v  Determine  psychological  mission  requirements  thai  predict 
success  ( and  failure)  and  use  to  screen  and  select. 
v  Metrics-based  selection  decisions  allow  flexibility  for 
changing  mission  conditions. 


l«i  m*  vm  vm  ira?  iwa  f?w  jwi  :wj  jwj  jwi  :<h»  we& 


.r  l».u",v  XI 1 1  H«i  ibJJ  Ml 1 


UbUtni'n-  h  Apprmtl  ta<  mU»  'ihu  ihittUm- 1.  uil'lnd  Lhii  m»nlikr  I  <W3  JS  J  J  »l « 
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❖  Prescreen  “Select  Out”  Variables 
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'Substance  Abuse  Issues  (self  or  family) 

'Previous  Suicide  Attempts 
'Poor  Marriage  Quality 
'Nightmares 

'Concerns  about  Past  Performance  Ratings 
'Family  Member  with  Needs/Problems 
'Anger  Issues 

'Takes  Tranquilizers  or  Anti -Depressants 

'Sleep  Problems 

'Dependent  with  Medical  Issues 

'Financial  Difficulties 

'Problems  with  the  Law 

'Mental  Health  Issues  (self  or  family) 

'Failed  to  Complete  a  Military  Course 

'Letter  of  Counseling/Reprimand,  Article  15,  or  Court 

Martial 


Distributor  Stalemert  A.  To-  ptDIio  release,  distibultf  is  dirtied .  Case  Number.  SSAB W-2Q 1 1  -4063. 25  Jul  20 1 1  3 
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Prescreen  Cost  Savings 
2005-Present 


Initial  Prescreen  -  MH  Tech, 
Psych,  Security  (n  =  1745) 


Combined  Savings  =  $2.2  million 


C, ill Dj  fii.i  Stele, 're  ,11  A  .■Vfi-T.'Al  lie  rile aa,  distiiJjli.1  is  ullirriled .  Case  Mu,fi>e,-.3ri\BW-3'Jl1-»63,;5  Jjl  20  1 1 
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Summary 
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Questions? 


v  Prescreen  Benefits 
v  Empirically  validated 
v  Consistent  with  research 
v  Improved  selection  of  special  duty  personnel 
v  Cost  savings 
v Applications 

v  Use  in  other  A&S  programs 
v  Use  in  selection  of  high-risk  operational 
personnel  (i.e.,  remotely  piloted  aircraft 
operators) 


Distiiteldn  Stele, re, 11  A.  ^ftauad  fa-piJelic  release,  distibulai  is  ullirriled.  Case  Nunriier.88ABW-201 1-4063,25  Jul  20 11 


Newly  dedicated  USAF SAM  facility 


Distibuldrt  Stele  rnertl  A  ^e^teved  fa-  piillid  release,  disVibulai  isuilirrtled.  Case  Nurtii)er.8SABW-201 1-4Q63, 25  Jul  3011  12 
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Benefits  of  Operational  Testing  and  Why  it’s  Important 

Maj  Charles  Morris  &  Maj  James  Weinstein,  AF  Medical  Evaluation  Support  Activity 

Educate  AFMS  personnel  on  the  importance  of  Operational  Testing  of  medical  equipment  and  systems 
(i.e.UTCs). 

The  Air  Force  Medical  Evaluation  Support  Activity  is  charged  with  conducting  Operational  Test  and  Evaluation 
(OT&E)  of  medical  and  IM/IT  equipment,  and  systems  by  the  Air  Force  Surgeon  General.  AFMESA  is  the 
AF’s  premier  medical  operational  test  activity.  AFMESA  testing  expertise  has  drawn  the  attention  of  other  DoD 
components,  and  government  agencies.  This  briefing  will  explain  how  AFMESA  conducts  OT&E,  why  it  is 
necessary,  how  it  differs  from  developmental  testing  and  current  trends  in  DoD  acquisition  that  are  driving 
changes  to  test  processes.  The  presentation  will  conclude  with  a  review  of  recent  test  programs  highlighting  the 
breadth  of  testing  environments,  scope  of  testing,  and  our  numerous  test  customers. 
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Headquarters  U.S.  Air  Force 
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Operational  Testing  in  Medical 

Acquisition 

Maj  Charles  Morris 
AFMSA  /  SG5T 
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Agenda 


Definition  of  Test  /  What  is  Test 
Where  Test  fits  into  Acquisitions 

Difference  between  Developmental  Testing  and  Operational 
Testing 

AFMESA: 
o  Mission/Org 
o  AFMESA  Advantage 
o  Authority 

o  Current  AFMESA  Customers 
AFMESA  Operational  Testing  Support 
Upcoming  /  Current  Activities 

Bottom  Line _ 

Integrity  -  Service  -  Excellence 
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Testing  Defined 


♦  Definition  of  Testing 

o  Testing  is  the  process  of  examining  and  operating  a  system 
or  systems  with  the  intent  of  finding  problems 

*  DOD  and  Service  Instructions  state  that  operational 
testing  will  test  a  product  for  effectiveness,  suitability,  and 
survivability 

o  The  actual  system  elements  of  what  fits  into  each  of  these 
metrics  are  application  and  system  dependent 

o  The  test  methods  used  to  test  each  of  these  metrics  are 
also  application  and  system  dependent 
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Testing  -  Where  We  Fit  in  the  Acquisitions 
Framework  for  Modernization 
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Pre-  Materiel  Technology  Engineerings.  Production  & 

Acq.  Solution  Developme  Manufacturing  Deployment 

Analysis  nt  Development 

k  SG9 

Program  &  Thrust  Area  Management 

Innovation  /  Research  and  Development 

Planning  Requirement  SG5  Costing  Program  Support, 

&  Development  &  Contracting  &  Budget 

Investing  &  Validation  Testing  Execution 

Operations 
&  Support 

SG6 

IMIT  Portfolio  &  Program  Management, 

Information  Assurance  /  Enterprise  Architecture 

SG5T 

Market  High  Earfy  Operational  Sustainment  and 

Surveillance  Performance  Operational  Testing  Force  Development 

/  Research  Teams  Assessments  Evaluation 
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DT(PMO)  vs.  OT(AFMESA) 


U  S.  *IR  FORCE 


Cotnro3kd  by  Fro§r»m 
Oac-OneTewt 


TWimiwi,  Experienced  Operators 

Firegge  Perfosmarae  Objectives  uud 
TBroeslioMI  Measorsmeist 
Tmto  Spmfkstiaa 
Des’dlcxpciectTesi  /fctide 


Users  Recemdy  TVamedl  turn  Equiijpamient 


Tfestto  &eqtajresn*itts 
Production!  Ready  ProdkiC 
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❖ 

U.S.fllR  FORCE 


What  is  Tested 


On  a  broad  scale,  operational  testing  focuses 
on  testing  and  measuring  a  product’s: 


i  Availability 
i  Compatibility 
i  Learnability 
i  Portability 
i  Reliability 
i  Supportability 
i  Suitability 


i  Capability 
i  Interoperability 
i  Maintainability 
i  Recoverability 
i  Usability 
i  Survivability 
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W  Report  of  the  Defense  Science 
.  Board  Task  Force  on  DT&E 

•  Findings 

o  IOT&E  failure  rate  (50%)  suggest  deficiencies  in  DT&E 
processes 

o  Suitability  failures  are  increasing 

•  Recommendations 

o  Integrate  RAM  in  system  development,  as  a  contractual 
requirement 

o  Improve  gov't  involvement  &  oversight  in  DT  -  access  to  test 
data 

o  Address  RAM  at  OTRR 

o  Integrated  DT/OT  to  share  resources  and  data 

o  Perform  detailed  risk  assessment  with  COTS  products 

•  Recent  DOT&E  memorandum  on  RAM  stresses  similar  issues 

o  Improved  RAM  decreases  Life  Cycle  Costs  and  reduces 
demand  on  logistics  systems 
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Relative  Cost  of 
Software  Fault  Propagation 
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U.S.  AIRFORCE 


Site  Overview 


Only  Dedicated  Medical  Operational  Testing  (OT)  Organization  in 
the  AFMS 

Stood  up  2001  per  AF/SG 
Fully  functional  EMEDS  +10 
31  Personnel 
Key  Staff 

o  Chief  -  Maj  James  Weinstei 
o  DO -Mr.  Jim  Sylvester 
o  Superintendent  -  SMSgt  Jason  Read 
GSU  Operation 

o  Fort  Detrick,  MD:  Medical  Systems,  Equipment,  UTCs 
o  Port  San  Antonio,  TX:  IM/IT  Systems 

Integrity  -  Service  -  Excellence 


U.S.  AIR  FORCE 


AFMESA /AFMISTB 


U.S.  AIRFORCE 


AFMESA  Advantage 


■  Two  DoD-Unique  Sites 
o  Ft  Detrick  Test  Site 

■  Established  in  2002 

-  14  Acres,  44,000  ft2  Test  Pad 

-  EMEDS  +10  /  Patient  Staging  UTC 

■  Biomedical  Equipment  Testing  Facility 

■  Unique  Personnel  Mix 

o  Medical  Information  Systems  Test  Bed,  Port  San  Antonio 

■  Established  in  2005 

■  DIACAP  Certificate  in  Late  June 

♦  OT  w/o  Borders,  Secured  Dedicated  VPN 

■  TATRC  CDE  Capability  in  2012 

♦  Able  to  Test  Systems  Before  Going  Live 
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Current  AFMESA  Customers 


U.S.  AIR  FORCE 
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U.S.  AIR  FORCE 


AFMESA  Tests  Throughout  the 
AF  and  DoD  Acquisition  Process 
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u.s.  AIR  FORCE 


Market  Research- 
Early  OT  Involvement 


Operational  Testing: 
AFSOC  Support 


Collect  and  Analyze  Information  about  Capabilities  within  the 
Market  to  Satisfy  Customer  Requirements  and  Needs 
Two  Main  Activities 

o  Market  Surveillance  (Practices.  Trends.  Technology  Development) 


Joint  Medical  Distant  SuppA 
and  Evacuation  ( JMO  SE)  1 
JCTD  Support 


Multi-Lingual  Interface  Device 

•Language  and  Dialect 
•  Do  s !  Don'ts  for  patient  CLiturel 
•Patient  inputs  symptoms  /  histto  y < 


M 


■  Quick  Look,  Strategic  View 

o  Market  Investigation  (In-Depth  Research.  Requires  a  Team 
Approach) 

-  In-depth  Look,  Tactical  View 


•Airarop  of  critical  combat  casualty  care 

suppon 

packages 

•Ir  DnewithAFs  _omt  P  east  on  Airdrop 
System 

iJPADS)  program 
•New  Development  ana  Concept 
■Kificro  Package  -  500  lbs 
•Ultra  Package  -250  lbs 
•72  hours  Ufa  Sustaining 
•Different  Configurations 
•Within  50  Meters 

EXPEDITIONARY  MEDICINE 


EXPEDITIONARY  MEDICINE 


CASEVAC  Electrical  Power  DaTTluTd 


•Man-portable  alternate  powesr 
sources 

•To  support  toward  combat  tx 1 
hosp  it  airtransport 


EXPEDITIONARY  MEDICINE 


Taking  Medical  Innovations  From  Concept  To  Field 
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Bottom  Line 


US.OIRFORCE 


Message  From  the  Boss 


•  Unique  Personnel  Mix  and  Medical  Experiences 
o  Easier  to  Make  Testers  Than  Clinicians 

•  Independent  and  Rapid 

•  Test  Early  and  Often 

.  Focus  on  Reliability,  Availability,  Maintainability, 
Sustainability,  and  Life  Cycle  Management 
o  Supporting  sustainability  and  maintainability 
(better  products  at  lower  cost)— the  Return  on 
Investment  (ROI)  for  engaging  test 

•  The  Key  for  a  Successful  Transition  from  Testing  to 
Field  is  Training  and  Technology  Adoption 


AFMESA  provides  the  Air  Force  Medical  Service,  DoD,  and 
our  Joint  partners  with  world  class  operational  testing  and 
evaluation  capabilities.  This  critical  asset  is  a  force  enabler 
and  a  powerful  risk  management  tool.  Test  is  the  conscience 
of  acquisition  -  the  final  arbiter  of  truth  -  did  we  build  the 
right  thing  for  our  warfighters? 

-Brig  Gen  James  Carroll,  AFMSA/CC 
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Operational  Testing  - 
“The  Right  Stuff” 
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U.S.AIWFORCE 


Questions 


AFMESA  Contact  Information 

http:  //af  mesa  .ft-detrick.  af.  m  il 
DSN  343-8600 
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